[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:53:28 dist-test-slave-m39l krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:53:28 dist-test-slave-m39l krb5kdc[399](info): set up 2 sockets
Sep 28 04:53:28 dist-test-slave-m39l krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 04:53:29.495054 392 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:53:29 dist-test-slave-m39l krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:53:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:53:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:53:29.627307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44353
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:29.715742 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:29.715970 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:29.715997 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:29.717219 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:53:29.717268 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:29.717283 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:29.717296 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:53:29.717309 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:53:29.718904 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:29.719187 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:29.719450 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:29.724730 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:29.724743 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:29.724705 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:29.724939 412 server_base.cc:1034] running on GCE node
I20260928 04:53:29.725229 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:29.725515 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:29.726650 412 hybrid_clock.cc:648] HybridClock initialized: now 1790571209726635 us; error 32 us; skew 500 ppm
Sep 28 04:53:29 dist-test-slave-m39l krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:29.729375 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:53:29.729887 412 webserver.cc:460] Webserver started at http://127.0.98.62:33343/ using document root <none> and password file <none>
I20260928 04:53:29.730116 412 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:29.730181 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:29.730333 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:29.731493 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "732e9128b0474225b3fa45ea9c46e550"
format_stamp: "Formatted at 2026-09-28 04:53:29 on dist-test-slave-m39l"
server_key: "60399f3236aa0a4b57f5c520a2dcac0b"
server_key_iv: "07958d5d132f56088cf780f91481388e"
server_key_version: "encryptionkey@0"
I20260928 04:53:29.731866 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "732e9128b0474225b3fa45ea9c46e550"
format_stamp: "Formatted at 2026-09-28 04:53:29 on dist-test-slave-m39l"
server_key: "60399f3236aa0a4b57f5c520a2dcac0b"
server_key_iv: "07958d5d132f56088cf780f91481388e"
server_key_version: "encryptionkey@0"
I20260928 04:53:29.733385 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:53:29.734216 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:29.734411 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:29.734550 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "732e9128b0474225b3fa45ea9c46e550"
format_stamp: "Formatted at 2026-09-28 04:53:29 on dist-test-slave-m39l"
server_key: "60399f3236aa0a4b57f5c520a2dcac0b"
server_key_iv: "07958d5d132f56088cf780f91481388e"
server_key_version: "encryptionkey@0"
I20260928 04:53:29.734643 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:29.758301 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:29.760959 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:29.761147 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:29.765520 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44353
I20260928 04:53:29.765547 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44353 every 8 connection(s)
I20260928 04:53:29.765955 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:53:29.766916 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:29.769237 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550: Bootstrap starting.
I20260928 04:53:29.769856 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:29.770144 480 log.cc:824] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:29.771034 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550: No bootstrap required, opened a new log
I20260928 04:53:29.772681 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732e9128b0474225b3fa45ea9c46e550" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44353 } }
I20260928 04:53:29.772889 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:29.772940 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732e9128b0474225b3fa45ea9c46e550, State: Initialized, Role: FOLLOWER
I20260928 04:53:29.773113 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732e9128b0474225b3fa45ea9c46e550" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44353 } }
I20260928 04:53:29.773219 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:53:29.773265 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:53:29.773274 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 412
I20260928 04:53:29.773319 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:29.773407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:53:29.773943 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732e9128b0474225b3fa45ea9c46e550" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44353 } }
I20260928 04:53:29.774061 480 leader_election.cc:302] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 732e9128b0474225b3fa45ea9c46e550; no voters:
I20260928 04:53:29.774206 392 external_mini_cluster.cc:1428] Setting key 4a13b5181c8020617ddfef0a88f68621
I20260928 04:53:29.774271 480 leader_election.cc:288] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:53:29.774298 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:29.774410 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550 [term 1 LEADER]: Becoming Leader. State: Replica: 732e9128b0474225b3fa45ea9c46e550, State: Running, Role: LEADER
I20260928 04:53:29.774525 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732e9128b0474225b3fa45ea9c46e550" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44353 } }
I20260928 04:53:29.774569 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:29.775197 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "732e9128b0474225b3fa45ea9c46e550" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732e9128b0474225b3fa45ea9c46e550" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44353 } } }
I20260928 04:53:29.775290 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:29.775426 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 732e9128b0474225b3fa45ea9c46e550. Latest consensus state: current_term: 1 leader_uuid: "732e9128b0474225b3fa45ea9c46e550" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732e9128b0474225b3fa45ea9c46e550" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44353 } } }
I20260928 04:53:29.775488 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550 [sys.catalog]: This master's current role is: LEADER
W20260928 04:53:29.776686 500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:53:29.776755 500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:53:29.778684 493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:53:29.778892 493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:53:29.780761 493 catalog_manager.cc:1347] Generated new cluster ID: dcc6b8c249494edd9708de372182203a
I20260928 04:53:29.780831 493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:53:29 dist-test-slave-m39l krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:29.791870 492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:29.775248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.0.98.62:44353)
0928 04:53:29.775783 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:29.775794 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:29.778265 (+ 2471us) server_negotiation.cc:407] Connection header received
0928 04:53:29.778370 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:29.778375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:29.778435 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:29.778629 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:29.780978 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:29.782039 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:29.783386 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:29.783882 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:29.784026 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:29.787832 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:29.787838 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:29.787838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:29.787858 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:29.789759 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:29.790253 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:29.790257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:29.790257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:29.790361 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:29.790687 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:29.790688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:29.790689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:29.790959 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:29.791129 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:29.791437 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 04:53:29.791588 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":93,"threads_started":1}
I20260928 04:53:29.793339 493 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:53:29.793891 493 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:53:29.804567 493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 732e9128b0474225b3fa45ea9c46e550: Generated new TSK 0
I20260928 04:53:29.804803 493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:53:29.834264 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44353
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:29.920083 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:29.920298 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:29.920324 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:29.921512 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:29.921555 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:29.921583 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:53:29.923178 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44353
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:29.923440 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:29.923691 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:29.929142 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:29.929147 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:29.929201 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:29.929360 506 server_base.cc:1034] running on GCE node
I20260928 04:53:29.929546 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:29.929785 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:29.930923 506 hybrid_clock.cc:648] HybridClock initialized: now 1790571209930908 us; error 28 us; skew 500 ppm
Sep 28 04:53:29 dist-test-slave-m39l krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:29.933475 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:53:29.933990 506 webserver.cc:460] Webserver started at http://127.0.98.1:37199/ using document root <none> and password file <none>
I20260928 04:53:29.934216 506 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:29.934281 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:29.934427 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:29.935544 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "306644b86b6b44c0bfa7a22fd88c8020"
format_stamp: "Formatted at 2026-09-28 04:53:29 on dist-test-slave-m39l"
server_key: "b5ca355876dfee73f4cf3dfa98f6d61c"
server_key_iv: "f22fc4c031145230c74ac929ca45b70c"
server_key_version: "encryptionkey@0"
I20260928 04:53:29.935922 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "306644b86b6b44c0bfa7a22fd88c8020"
format_stamp: "Formatted at 2026-09-28 04:53:29 on dist-test-slave-m39l"
server_key: "b5ca355876dfee73f4cf3dfa98f6d61c"
server_key_iv: "f22fc4c031145230c74ac929ca45b70c"
server_key_version: "encryptionkey@0"
I20260928 04:53:29.937772 506 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 04:53:29.938542 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:29.938737 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:29.938832 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "306644b86b6b44c0bfa7a22fd88c8020"
format_stamp: "Formatted at 2026-09-28 04:53:29 on dist-test-slave-m39l"
server_key: "b5ca355876dfee73f4cf3dfa98f6d61c"
server_key_iv: "f22fc4c031145230c74ac929ca45b70c"
server_key_version: "encryptionkey@0"
I20260928 04:53:29.938911 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:29.953282 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:29.955602 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:29.955821 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:29.956179 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:29.956565 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:29.956619 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:29.956661 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:29.956694 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:29.963088 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37239
I20260928 04:53:29.963126 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37239 every 8 connection(s)
I20260928 04:53:29.963505 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:53:29.969550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 506
I20260928 04:53:29.969642 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:53:29.969805 392 external_mini_cluster.cc:1428] Setting key 9fe01f725cf5c459dee517d0b2dcfc36
Sep 28 04:53:29 dist-test-slave-m39l krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:29.974133 492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:29.964322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57961 (local address 127.0.98.62:44353)
0928 04:53:29.964385 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:29.964387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:29.964568 (+ 181us) server_negotiation.cc:407] Connection header received
0928 04:53:29.965120 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:29.965122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:29.965138 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:29.965215 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:29.966260 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:29.966851 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:29.967869 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:29.968074 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:29.968161 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:29.970874 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:29.970876 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:29.970877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:29.970885 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:29.972509 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:29.972994 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:29.972996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:29.972996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:29.973086 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:29.973364 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:29.973365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:29.973366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:29.973503 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:29.973586 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:29.973881 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 04:53:29.974034 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 04:53:29.974349 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:29.964257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44353 (local address 127.0.98.1:57961)
0928 04:53:29.964439 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:29.964484 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:29.965011 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:29.965295 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:29.965304 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:29.965913 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:29.966119 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:29.966125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:29.966941 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:29.966942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:29.967775 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:29.967777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:29.968171 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:29.968172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:29.968300 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:29.968864 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:29.968872 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:29.970748 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:29.972622 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:29.972632 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:29.972635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:29.972900 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:29.973159 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:29.973160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:29.973160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:29.973289 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:29.973576 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:29.973583 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:29.973740 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 04:53:29.973859 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 04:53:29.974080 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":66,"threads_started":1}
I20260928 04:53:29.974937 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44353
I20260928 04:53:29.975083 635 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:29.975312 635 heartbeater.cc:511] Master 127.0.98.62:44353 requested a full tablet report, sending...
I20260928 04:53:29.976001 444 ts_manager.cc:194] Registered new tserver with Master: 306644b86b6b44c0bfa7a22fd88c8020 (127.0.98.1:37239)
I20260928 04:53:29.976971 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57961
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:53:29.998656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44353
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:30.085762 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:30.085975 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:30.086015 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:30.087194 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:30.087237 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:30.087265 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:53:30.088660 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44353
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:30.088850 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:30.089136 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:30.095094 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:30.095264 640 server_base.cc:1034] running on GCE node
W20260928 04:53:30.095160 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:30.095126 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:30.095583 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:30.095814 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:30.096961 640 hybrid_clock.cc:648] HybridClock initialized: now 1790571210096922 us; error 61 us; skew 500 ppm
Sep 28 04:53:30 dist-test-slave-m39l krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:30.100117 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:53:30.100622 640 webserver.cc:460] Webserver started at http://127.0.98.2:37879/ using document root <none> and password file <none>
I20260928 04:53:30.100862 640 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:30.100930 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:30.101078 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:30.102234 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "3cc42d93b433444ba34ef7c58464ba15"
format_stamp: "Formatted at 2026-09-28 04:53:30 on dist-test-slave-m39l"
server_key: "3f0813002888f2ec9ef6e390904eb371"
server_key_iv: "5d3320d5d7d3cfbb2e057f15e8d42462"
server_key_version: "encryptionkey@0"
I20260928 04:53:30.102638 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3cc42d93b433444ba34ef7c58464ba15"
format_stamp: "Formatted at 2026-09-28 04:53:30 on dist-test-slave-m39l"
server_key: "3f0813002888f2ec9ef6e390904eb371"
server_key_iv: "5d3320d5d7d3cfbb2e057f15e8d42462"
server_key_version: "encryptionkey@0"
I20260928 04:53:30.104029 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:53:30.104786 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:30.104949 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:53:30.105059 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "3cc42d93b433444ba34ef7c58464ba15"
format_stamp: "Formatted at 2026-09-28 04:53:30 on dist-test-slave-m39l"
server_key: "3f0813002888f2ec9ef6e390904eb371"
server_key_iv: "5d3320d5d7d3cfbb2e057f15e8d42462"
server_key_version: "encryptionkey@0"
I20260928 04:53:30.105131 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:30.131598 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:30.133765 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:30.133931 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:30.134176 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:30.134573 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:30.134625 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:30.134667 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:30.134691 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:30.141654 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38323
I20260928 04:53:30.141711 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38323 every 8 connection(s)
I20260928 04:53:30.142089 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:53:30.144174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 640
I20260928 04:53:30.144291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:53:30.144452 392 external_mini_cluster.cc:1428] Setting key 1522392a02a2d8c6b4dcc9baba64995b
Sep 28 04:53:30 dist-test-slave-m39l krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:30.152654 492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:30.142860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38127 (local address 127.0.98.62:44353)
0928 04:53:30.142935 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:30.142937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:30.143151 (+ 214us) server_negotiation.cc:407] Connection header received
0928 04:53:30.143672 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:30.143674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:30.143686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:30.143733 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:30.144801 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:30.145311 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:30.146463 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:30.146736 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:30.146800 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:30.149497 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:30.149499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:30.149499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:30.149507 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:30.151011 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:30.151554 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:30.151556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:30.151556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:30.151612 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:30.151912 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:30.151912 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:30.151912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:30.152039 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:30.152100 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:30.152481 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 04:53:30.152584 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 04:53:30.152931 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:30.142842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44353 (local address 127.0.98.2:38127)
0928 04:53:30.143032 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:30.143069 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:30.143573 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:30.143816 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:30.143821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:30.144456 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:30.144701 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:30.144706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:30.145402 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:30.145403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:30.146372 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:30.146375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:30.146822 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:30.146824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:30.146981 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:30.147541 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:30.147552 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:30.149364 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:30.151107 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:30.151117 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:30.151120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:30.151450 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:30.151690 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:30.151691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:30.151691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:30.151839 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:30.152109 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:30.152116 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:30.152320 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 04:53:30.152452 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 04:53:30.152639 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":104,"threads_started":1}
I20260928 04:53:30.153470 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44353
I20260928 04:53:30.153592 769 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:30.153820 769 heartbeater.cc:511] Master 127.0.98.62:44353 requested a full tablet report, sending...
I20260928 04:53:30.154254 444 ts_manager.cc:194] Registered new tserver with Master: 3cc42d93b433444ba34ef7c58464ba15 (127.0.98.2:38323)
I20260928 04:53:30.154680 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:53:30.174296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44353
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:30.257704 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:30.257946 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:30.257974 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:30.259200 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:30.259243 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:30.259274 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:53:30.260721 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44353
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:30.260932 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:30.261125 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:30.266618 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:30.266639 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:30.266618 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:30.266659 774 server_base.cc:1034] running on GCE node
I20260928 04:53:30.267072 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:30.267295 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:30.268431 774 hybrid_clock.cc:648] HybridClock initialized: now 1790571210268419 us; error 30 us; skew 500 ppm
Sep 28 04:53:30 dist-test-slave-m39l krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:30.271057 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:53:30.271544 774 webserver.cc:460] Webserver started at http://127.0.98.3:34931/ using document root <none> and password file <none>
I20260928 04:53:30.271780 774 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:30.271827 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:30.271971 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:30.273079 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "eb788f481d714b9586698a09699f3cce"
format_stamp: "Formatted at 2026-09-28 04:53:30 on dist-test-slave-m39l"
server_key: "499139af0bf6bdf5057ece08f7c1ea09"
server_key_iv: "82c427e86b408405996ec3965ca8d050"
server_key_version: "encryptionkey@0"
I20260928 04:53:30.273452 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eb788f481d714b9586698a09699f3cce"
format_stamp: "Formatted at 2026-09-28 04:53:30 on dist-test-slave-m39l"
server_key: "499139af0bf6bdf5057ece08f7c1ea09"
server_key_iv: "82c427e86b408405996ec3965ca8d050"
server_key_version: "encryptionkey@0"
I20260928 04:53:30.274996 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:53:30.275842 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:30.276060 774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:30.276173 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "eb788f481d714b9586698a09699f3cce"
format_stamp: "Formatted at 2026-09-28 04:53:30 on dist-test-slave-m39l"
server_key: "499139af0bf6bdf5057ece08f7c1ea09"
server_key_iv: "82c427e86b408405996ec3965ca8d050"
server_key_version: "encryptionkey@0"
I20260928 04:53:30.276259 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:30.286732 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:30.289565 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:30.289767 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:30.290023 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:30.290359 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:30.290413 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:30.290457 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:30.290510 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:30.298188 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33493
I20260928 04:53:30.298228 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33493 every 8 connection(s)
I20260928 04:53:30.298656 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:53:30.299666 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 774
I20260928 04:53:30.299752 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:53:30.299921 392 external_mini_cluster.cc:1428] Setting key 63bb138521dc97df2f54e422ddebc023
Sep 28 04:53:30 dist-test-slave-m39l krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:30.309334 492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:30.299431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36929 (local address 127.0.98.62:44353)
0928 04:53:30.299479 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:30.299481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:30.299785 (+ 304us) server_negotiation.cc:407] Connection header received
0928 04:53:30.300435 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:30.300437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:30.300449 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:30.300504 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:30.301733 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:30.302387 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:30.303521 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:30.303725 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:30.303795 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:30.306334 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:30.306336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:30.306336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:30.306345 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:30.308124 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:30.308490 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:30.308491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:30.308492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:30.308547 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:30.308746 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:30.308746 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:30.308746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:30.308855 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:30.308906 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:30.309189 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 04:53:30.309260 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:53:30.309576 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:30.299422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44353 (local address 127.0.98.3:36929)
0928 04:53:30.299630 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:30.299685 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:30.300349 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:30.300584 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:30.300593 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:30.301340 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:30.301603 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:30.301608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:30.302540 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:30.302542 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:30.303409 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:30.303411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:30.303813 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:30.303814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:30.303916 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:30.304632 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:30.304643 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:30.306231 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:30.308205 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:30.308211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:30.308214 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:30.308438 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:30.308603 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:30.308603 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:30.308604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:30.308705 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:30.308904 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:30.308908 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:30.309081 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 04:53:30.309185 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 04:53:30.309316 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":102,"threads_started":1}
I20260928 04:53:30.310094 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44353
I20260928 04:53:30.310192 903 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:30.310366 903 heartbeater.cc:511] Master 127.0.98.62:44353 requested a full tablet report, sending...
I20260928 04:53:30.310798 444 ts_manager.cc:194] Registered new tserver with Master: eb788f481d714b9586698a09699f3cce (127.0.98.3:33493)
I20260928 04:53:30.311254 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36929
I20260928 04:53:30.312129 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 04:53:30 dist-test-slave-m39l krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571210, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 04:53:30 dist-test-slave-m39l krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571210, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:53:30.337335 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:30.328702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44582 (local address 127.0.98.1:37239)
0928 04:53:30.328883 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:30.328888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:30.329021 (+ 133us) server_negotiation.cc:407] Connection header received
0928 04:53:30.329059 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:30.329062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:30.329081 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:30.329179 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:30.329819 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:30.330712 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:30.331460 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:30.331811 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:30.331877 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:30.334218 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:30.334228 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:30.334228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:30.334237 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:30.335993 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:30.336409 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:30.336413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:30.336413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:30.336491 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:30.336693 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:30.336694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:30.336695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:30.336852 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:30.337020 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:30.337116 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 04:53:30.337206 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
W20260928 04:53:30.337757 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44582
Sep 28 04:53:30 dist-test-slave-m39l krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 04:53:30 dist-test-slave-m39l krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:53:30.363473 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:30.355160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.0.98.1:37239)
0928 04:53:30.355223 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:30.355225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:30.355299 (+ 74us) server_negotiation.cc:407] Connection header received
0928 04:53:30.355395 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:30.355396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:30.355414 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:30.355476 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:30.356117 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:30.356606 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:30.357348 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:30.357568 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:30.357655 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:30.360331 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:30.360334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:30.360334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:30.360348 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:30.362132 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:30.362599 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:30.362601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:30.362601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:30.362669 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:30.362891 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:30.362893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:30.362893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:30.363012 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:30.363156 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:30.363254 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 04:53:30.363357 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:53:30.364164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 506
I20260928 04:53:30.369226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 640
I20260928 04:53:30.375946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 774
I20260928 04:53:30.381183 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 412
2026-09-28T04:53:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1975 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:53:30 dist-test-slave-m39l krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:53:30 dist-test-slave-m39l krb5kdc[917](info): set up 2 sockets
Sep 28 04:53:30 dist-test-slave-m39l krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260928 04:53:31.470324 392 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:53:31 dist-test-slave-m39l krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:53:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:53:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:53:31.599257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38083
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:31.683013 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:31.683317 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:31.683373 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:31.684744 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:53:31.684831 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:31.684856 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:31.684890 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:53:31.684934 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:53:31.686787 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:31.687105 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:31.687361 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:31.692575 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:31.692798 930 server_base.cc:1034] running on GCE node
W20260928 04:53:31.692605 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:31.692556 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:31.693245 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:31.693578 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:31.694743 930 hybrid_clock.cc:648] HybridClock initialized: now 1790571211694722 us; error 43 us; skew 500 ppm
Sep 28 04:53:31 dist-test-slave-m39l krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:31.701654 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:53:31.702250 930 webserver.cc:460] Webserver started at http://127.0.98.62:45133/ using document root <none> and password file <none>
I20260928 04:53:31.702587 930 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:31.702668 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:31.702811 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:31.704067 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "b16126262fc1419bb26008816faf12cb"
format_stamp: "Formatted at 2026-09-28 04:53:31 on dist-test-slave-m39l"
server_key: "e019ea2d2ad154ab10fdd6086fb4e06f"
server_key_iv: "f5bc4d6af66c676519f146de04b64ea7"
server_key_version: "encryptionkey@0"
I20260928 04:53:31.704442 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "b16126262fc1419bb26008816faf12cb"
format_stamp: "Formatted at 2026-09-28 04:53:31 on dist-test-slave-m39l"
server_key: "e019ea2d2ad154ab10fdd6086fb4e06f"
server_key_iv: "f5bc4d6af66c676519f146de04b64ea7"
server_key_version: "encryptionkey@0"
I20260928 04:53:31.705849 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:53:31.706826 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:31.707175 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:53:31.707288 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "b16126262fc1419bb26008816faf12cb"
format_stamp: "Formatted at 2026-09-28 04:53:31 on dist-test-slave-m39l"
server_key: "e019ea2d2ad154ab10fdd6086fb4e06f"
server_key_iv: "f5bc4d6af66c676519f146de04b64ea7"
server_key_version: "encryptionkey@0"
I20260928 04:53:31.707351 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:31.740337 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:31.743003 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:31.743139 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:31.747435 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38083
I20260928 04:53:31.747471 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38083 every 8 connection(s)
I20260928 04:53:31.747892 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:53:31.748803 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:31.751217 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb: Bootstrap starting.
I20260928 04:53:31.751761 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:31.752015 998 log.cc:824] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:31.752718 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb: No bootstrap required, opened a new log
I20260928 04:53:31.754395 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16126262fc1419bb26008816faf12cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38083 } }
I20260928 04:53:31.754633 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:31.754662 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b16126262fc1419bb26008816faf12cb, State: Initialized, Role: FOLLOWER
I20260928 04:53:31.754804 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16126262fc1419bb26008816faf12cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38083 } }
I20260928 04:53:31.754889 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:53:31.754932 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:53:31.754984 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:31.755113 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 930
I20260928 04:53:31.755213 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:53:31.755391 392 external_mini_cluster.cc:1428] Setting key ca33c00700fb7e813ad7fc22459eca45
I20260928 04:53:31.755704 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16126262fc1419bb26008816faf12cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38083 } }
I20260928 04:53:31.755836 998 leader_election.cc:302] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b16126262fc1419bb26008816faf12cb; no voters:
I20260928 04:53:31.756313 998 leader_election.cc:288] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:53:31.756564 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:31.756574 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:31.756742 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb [term 1 LEADER]: Becoming Leader. State: Replica: b16126262fc1419bb26008816faf12cb, State: Running, Role: LEADER
I20260928 04:53:31.756847 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16126262fc1419bb26008816faf12cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38083 } }
I20260928 04:53:31.757203 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b16126262fc1419bb26008816faf12cb. Latest consensus state: current_term: 1 leader_uuid: "b16126262fc1419bb26008816faf12cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16126262fc1419bb26008816faf12cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38083 } } }
I20260928 04:53:31.757301 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:31.757467 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b16126262fc1419bb26008816faf12cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16126262fc1419bb26008816faf12cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38083 } } }
I20260928 04:53:31.757527 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:31.757642 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:53:31.758226 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:53:31.761112 1012 catalog_manager.cc:1347] Generated new cluster ID: 3b148ec8a061482a8212476cd5d65bb6
I20260928 04:53:31.761188 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:53:31 dist-test-slave-m39l krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:31.769248 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:31.755923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42172 (local address 127.0.98.62:38083)
0928 04:53:31.756171 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:31.756181 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:31.756245 (+ 64us) server_negotiation.cc:407] Connection header received
0928 04:53:31.756399 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:31.756410 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:31.756478 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:31.756668 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:31.757835 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:31.759487 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:31.760446 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:31.761095 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:31.761209 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:31.764364 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:31.764369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:31.764370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:31.764385 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:31.767433 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:31.767877 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:31.767881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:31.767882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:31.768020 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:31.768221 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:31.768222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:31.768223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:31.768612 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:31.768791 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:31.768866 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 04:53:31.769016 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":142,"threads_started":1}
I20260928 04:53:31.773680 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:53:31.774219 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:53:31.782330 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b16126262fc1419bb26008816faf12cb: Generated new TSK 0
I20260928 04:53:31.782605 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:53:31.819166 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38083
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:31.904639 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:31.904858 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:31.904886 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:31.906126 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:31.906176 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:31.906229 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:53:31.907810 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38083
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:31.908064 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:31.908321 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:53:31.913678 1024 server_base.cc:1034] running on GCE node
W20260928 04:53:31.913657 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:31.913631 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:31.913641 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:31.914080 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:31.914350 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:31.915550 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790571211915523 us; error 56 us; skew 500 ppm
Sep 28 04:53:31 dist-test-slave-m39l krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:31.919070 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:53:31.919623 1024 webserver.cc:460] Webserver started at http://127.0.98.1:35387/ using document root <none> and password file <none>
I20260928 04:53:31.919907 1024 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:31.919978 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:31.920125 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:31.921253 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "e92358262c0145c281b2b75d584f536c"
format_stamp: "Formatted at 2026-09-28 04:53:31 on dist-test-slave-m39l"
server_key: "f440b5b33bab7f4a7f610dd28878cc45"
server_key_iv: "913f9e0bd1646fc1226beee7b73616d4"
server_key_version: "encryptionkey@0"
I20260928 04:53:31.921645 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e92358262c0145c281b2b75d584f536c"
format_stamp: "Formatted at 2026-09-28 04:53:31 on dist-test-slave-m39l"
server_key: "f440b5b33bab7f4a7f610dd28878cc45"
server_key_iv: "913f9e0bd1646fc1226beee7b73616d4"
server_key_version: "encryptionkey@0"
I20260928 04:53:31.923210 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:53:31.924127 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:31.924329 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:31.924432 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "e92358262c0145c281b2b75d584f536c"
format_stamp: "Formatted at 2026-09-28 04:53:31 on dist-test-slave-m39l"
server_key: "f440b5b33bab7f4a7f610dd28878cc45"
server_key_iv: "913f9e0bd1646fc1226beee7b73616d4"
server_key_version: "encryptionkey@0"
I20260928 04:53:31.924541 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:31.954864 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:31.957291 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:31.957449 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:31.957700 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:31.958024 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:31.958077 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:31.958122 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:31.958148 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:31.964090 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46103
I20260928 04:53:31.964211 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46103 every 8 connection(s)
I20260928 04:53:31.964694 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:53:31 dist-test-slave-m39l krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:31.974797 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 1024
I20260928 04:53:31.974900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:53:31.975076 392 external_mini_cluster.cc:1428] Setting key de6a9f9911815560554b27f8a252e66f
I20260928 04:53:31.975095 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:31.965611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33913 (local address 127.0.98.62:38083)
0928 04:53:31.965671 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:31.965674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:31.965936 (+ 262us) server_negotiation.cc:407] Connection header received
0928 04:53:31.966653 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:31.966654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:31.966671 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:31.966734 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:31.967735 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:31.968269 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:31.969311 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:31.969540 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:31.969601 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:31.972044 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:31.972045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:31.972045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:31.972055 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:31.973545 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:31.973938 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:31.973938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:31.973939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:31.973991 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:31.974273 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:31.974274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:31.974274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:31.974380 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:31.974429 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:31.974790 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 04:53:31.974937 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:53:31.975270 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:31.965607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38083 (local address 127.0.98.1:33913)
0928 04:53:31.965783 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:31.965848 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:31.966365 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:31.966811 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:31.966815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:31.967404 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:31.967640 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:31.967646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:31.968349 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:31.968350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:31.969223 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:31.969226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:31.969638 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:31.969640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:31.969772 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:31.970403 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:31.970412 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:31.971933 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:31.973624 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:31.973630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:31.973632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:31.973871 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:31.974101 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:31.974102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:31.974102 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:31.974218 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:31.974431 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:31.974435 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:31.974613 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 04:53:31.974748 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 04:53:31.974942 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
I20260928 04:53:31.975907 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38083
I20260928 04:53:31.976008 1153 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:31.976207 1153 heartbeater.cc:511] Master 127.0.98.62:38083 requested a full tablet report, sending...
I20260928 04:53:31.976830 962 ts_manager.cc:194] Registered new tserver with Master: e92358262c0145c281b2b75d584f536c (127.0.98.1:46103)
I20260928 04:53:31.977541 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33913
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:53:32.002876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38083
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:32.090265 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:32.090550 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:32.090598 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:32.091827 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:32.091900 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:32.091946 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:53:32.093564 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38083
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:32.093875 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:32.094126 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:32.099799 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:32.099859 1158 server_base.cc:1034] running on GCE node
W20260928 04:53:32.099800 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:32.099805 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:32.100322 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:32.100608 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:32.101788 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790571212101756 us; error 36 us; skew 500 ppm
Sep 28 04:53:32 dist-test-slave-m39l krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:32.104864 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:53:32.105542 1158 webserver.cc:460] Webserver started at http://127.0.98.2:38445/ using document root <none> and password file <none>
I20260928 04:53:32.105800 1158 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:32.105870 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:32.106016 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:32.107163 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "58c4288b7e734dd692c446fe005f2901"
format_stamp: "Formatted at 2026-09-28 04:53:32 on dist-test-slave-m39l"
server_key: "d8efe98b556dafd7a883649892660bea"
server_key_iv: "ac4e7a778c5b5bafb580019800ec595e"
server_key_version: "encryptionkey@0"
I20260928 04:53:32.107553 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "58c4288b7e734dd692c446fe005f2901"
format_stamp: "Formatted at 2026-09-28 04:53:32 on dist-test-slave-m39l"
server_key: "d8efe98b556dafd7a883649892660bea"
server_key_iv: "ac4e7a778c5b5bafb580019800ec595e"
server_key_version: "encryptionkey@0"
I20260928 04:53:32.108928 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:53:32.109900 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:32.110142 1158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:32.110246 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "58c4288b7e734dd692c446fe005f2901"
format_stamp: "Formatted at 2026-09-28 04:53:32 on dist-test-slave-m39l"
server_key: "d8efe98b556dafd7a883649892660bea"
server_key_iv: "ac4e7a778c5b5bafb580019800ec595e"
server_key_version: "encryptionkey@0"
I20260928 04:53:32.110330 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:32.139904 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:32.142148 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:32.142331 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:32.142590 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:32.142910 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:32.142967 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:32.143008 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:32.143044 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:32.148747 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41059
I20260928 04:53:32.148847 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41059 every 8 connection(s)
I20260928 04:53:32.149155 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:53:32 dist-test-slave-m39l krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:32.158419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 1158
I20260928 04:53:32.158545 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:53:32.158731 392 external_mini_cluster.cc:1428] Setting key f2c5c3a17f4785fd82a94eb2b84c21c0
I20260928 04:53:32.159539 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:32.149838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59443 (local address 127.0.98.62:38083)
0928 04:53:32.149927 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:32.149929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:32.150111 (+ 182us) server_negotiation.cc:407] Connection header received
0928 04:53:32.150661 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:32.150662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:32.150674 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:32.150741 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:32.151731 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.152279 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.153337 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.153530 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.153600 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:32.156103 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:32.156105 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:32.156105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:32.156118 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:32.157775 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:32.158199 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:32.158201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:32.158201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:32.158266 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:32.158664 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:32.158666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:32.158666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:32.158826 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:32.158907 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:32.159316 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 04:53:32.159421 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 04:53:32.159699 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:32.149793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38083 (local address 127.0.98.2:59443)
0928 04:53:32.149948 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:32.149991 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:32.150550 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:32.150818 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:32.150821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:32.151405 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:32.151628 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:32.151632 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.152381 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.152382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:32.153249 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:32.153251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.153656 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.153657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:32.153782 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:32.154384 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:32.154392 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:32.155898 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:32.157863 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:32.157869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:32.157871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:32.158111 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:32.158347 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:32.158348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:32.158349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:32.158564 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:32.158889 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:32.158896 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:32.159148 (+ 252us) client_negotiation.cc:769] Sending connection context
0928 04:53:32.159252 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 04:53:32.159429 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260928 04:53:32.160249 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38083
I20260928 04:53:32.160370 1287 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:32.160565 1287 heartbeater.cc:511] Master 127.0.98.62:38083 requested a full tablet report, sending...
I20260928 04:53:32.161022 962 ts_manager.cc:194] Registered new tserver with Master: 58c4288b7e734dd692c446fe005f2901 (127.0.98.2:41059)
I20260928 04:53:32.161429 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59443
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:53:32.190894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38083
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:32.275071 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:32.275326 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:32.275367 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:32.276633 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:32.276721 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:32.276772 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:53:32.278369 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38083
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:32.278703 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:32.278967 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:32.284487 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:32.284446 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:32.284446 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:32.284487 1292 server_base.cc:1034] running on GCE node
I20260928 04:53:32.284873 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:32.285151 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:32.286358 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790571212286349 us; error 51 us; skew 500 ppm
Sep 28 04:53:32 dist-test-slave-m39l krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:32.289296 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:53:32.289825 1292 webserver.cc:460] Webserver started at http://127.0.98.3:45395/ using document root <none> and password file <none>
I20260928 04:53:32.290059 1292 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:32.290126 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:32.290288 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:32.291447 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "3dbbaa2ccd844213819b12009a9a2123"
format_stamp: "Formatted at 2026-09-28 04:53:32 on dist-test-slave-m39l"
server_key: "8bfbe6ee245199ebe7770e2a7116f71b"
server_key_iv: "e999b506b8d1ed778a76b82e7688b331"
server_key_version: "encryptionkey@0"
I20260928 04:53:32.291827 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3dbbaa2ccd844213819b12009a9a2123"
format_stamp: "Formatted at 2026-09-28 04:53:32 on dist-test-slave-m39l"
server_key: "8bfbe6ee245199ebe7770e2a7116f71b"
server_key_iv: "e999b506b8d1ed778a76b82e7688b331"
server_key_version: "encryptionkey@0"
I20260928 04:53:32.293262 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:53:32.294166 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:32.294371 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:32.294520 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "3dbbaa2ccd844213819b12009a9a2123"
format_stamp: "Formatted at 2026-09-28 04:53:32 on dist-test-slave-m39l"
server_key: "8bfbe6ee245199ebe7770e2a7116f71b"
server_key_iv: "e999b506b8d1ed778a76b82e7688b331"
server_key_version: "encryptionkey@0"
I20260928 04:53:32.294613 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:32.308964 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:32.311513 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:32.311685 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:32.311928 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:32.312243 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:32.312294 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:32.312335 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:32.312361 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:32.318339 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36803
I20260928 04:53:32.318377 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36803 every 8 connection(s)
I20260928 04:53:32.318764 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:53:32 dist-test-slave-m39l krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:32.326200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 1292
I20260928 04:53:32.326319 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:53:32.326562 392 external_mini_cluster.cc:1428] Setting key a1d1ccc40e7bb3c1cd5d24005b3cdd31
I20260928 04:53:32.329489 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:32.319566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37529 (local address 127.0.98.62:38083)
0928 04:53:32.319624 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:32.319626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:32.319837 (+ 211us) server_negotiation.cc:407] Connection header received
0928 04:53:32.320325 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:32.320326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:32.320335 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:32.320379 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:32.321352 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.321896 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.323182 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.323372 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.323437 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:32.325811 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:32.325813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:32.325813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:32.325820 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:32.327848 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:32.328349 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:32.328350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:32.328350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:32.328431 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:32.328689 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:32.328690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:32.328690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:32.328826 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:32.328891 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:32.329302 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 04:53:32.329388 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:53:32.329645 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:32.319540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38083 (local address 127.0.98.3:37529)
0928 04:53:32.319702 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:32.319746 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:32.320223 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:32.320437 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:32.320441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:32.321044 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:32.321242 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:32.321246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.322142 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.322143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:32.323082 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:32.323085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.323448 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.323449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:32.323559 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:32.324196 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:32.324205 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:32.325693 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:32.327943 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:32.328022 (+ 79us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:32.328025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:32.328284 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:32.328502 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:32.328502 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:32.328502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:32.328607 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:32.328916 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:32.328923 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:32.329141 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 04:53:32.329248 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 04:53:32.329400 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260928 04:53:32.330158 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38083
I20260928 04:53:32.330253 1421 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:32.330464 1421 heartbeater.cc:511] Master 127.0.98.62:38083 requested a full tablet report, sending...
I20260928 04:53:32.330957 962 ts_manager.cc:194] Registered new tserver with Master: 3dbbaa2ccd844213819b12009a9a2123 (127.0.98.3:36803)
I20260928 04:53:32.331367 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37529
I20260928 04:53:32.338979 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:53:32.347138 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:32.340616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.0.98.62:38083)
0928 04:53:32.340697 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:32.340699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:32.340766 (+ 67us) server_negotiation.cc:407] Connection header received
0928 04:53:32.340855 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:32.340856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:32.340866 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:32.340943 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:32.341538 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.342059 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.342889 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.343091 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.343172 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:32.344524 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:32.344526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:32.344526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:32.344535 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:32.345828 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:32.346243 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:32.346245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:32.346245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:32.346311 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:32.346528 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:32.346529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:32.346530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:32.346701 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:32.346810 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:32.346925 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 04:53:32.347017 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 04:53:32.349318 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:53:32.350940 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:53:32.357774 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:32.353154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41059 (local address 127.0.0.1:60092)
0928 04:53:32.353875 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:32.353881 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:32.353940 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:32.354214 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:32.354215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:32.354218 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:32.354270 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:32.354272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.355237 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.355237 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:32.356499 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:32.356501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.357520 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.357521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:32.357579 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:32.357585 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 04:53:32.357634 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 04:53:32.357667 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":100,"threads_started":1}
I20260928 04:53:32.357863 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:32.353548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59936 (local address 127.0.98.1:46103)
0928 04:53:32.353698 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:32.353701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:32.353782 (+ 81us) server_negotiation.cc:407] Connection header received
0928 04:53:32.353876 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:32.353878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:32.353891 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:32.353970 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:32.354225 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.355403 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.356673 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.357262 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.357364 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:32.357429 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:32.357729 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 04:53:32.357772 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260928 04:53:32.357981 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:32.353149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46103 (local address 127.0.0.1:59936)
0928 04:53:32.353706 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:32.353738 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:32.353800 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:32.353989 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:32.353993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:32.353996 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:32.354083 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:32.354125 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.355508 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.355509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:32.356597 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:32.356600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.357366 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.357367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:32.357456 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:32.357471 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 04:53:32.357553 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 04:53:32.357617 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":1,"thread_start_us":52,"threads_started":1}
I20260928 04:53:32.358074 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:32.353163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36803 (local address 127.0.0.1:40334)
0928 04:53:32.354016 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:32.354024 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:32.354066 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:32.354321 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:32.354322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:32.354324 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:32.354356 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:32.354358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.355594 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.355595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:32.356514 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:32.356516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.357617 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.357618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:32.357683 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:32.357688 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:53:32.357740 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 04:53:32.357782 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":91,"threads_started":1}
I20260928 04:53:32.358242 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:32.353391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.0.98.3:36803)
0928 04:53:32.353556 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:32.353561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:32.354126 (+ 565us) server_negotiation.cc:407] Connection header received
0928 04:53:32.354170 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:32.354173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:32.354187 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:32.354255 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:32.354434 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.355509 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.356653 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.357440 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.357521 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:32.357572 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:32.357822 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 04:53:32.358019 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260928 04:53:32.357859 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:32.353365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.0.98.2:41059)
0928 04:53:32.353556 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:32.353561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:32.353943 (+ 382us) server_negotiation.cc:407] Connection header received
0928 04:53:32.354018 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:32.354021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:32.354048 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:32.354125 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:32.354356 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.355139 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.356646 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.357440 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.357517 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:32.357572 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:32.357669 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 04:53:32.357749 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":77,"threads_started":1}
I20260928 04:53:32.358978 1087 tablet_service.cc:1467] Processing CreateTablet for tablet d23311cfc0484c4dbd23c602b4f99af0 (DEFAULT_TABLE table=test-table [id=c782cf9147f644259b78aef7d3e49418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:32.358978 1355 tablet_service.cc:1467] Processing CreateTablet for tablet d23311cfc0484c4dbd23c602b4f99af0 (DEFAULT_TABLE table=test-table [id=c782cf9147f644259b78aef7d3e49418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:32.359097 1221 tablet_service.cc:1467] Processing CreateTablet for tablet d23311cfc0484c4dbd23c602b4f99af0 (DEFAULT_TABLE table=test-table [id=c782cf9147f644259b78aef7d3e49418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:32.359331 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d23311cfc0484c4dbd23c602b4f99af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:32.359333 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d23311cfc0484c4dbd23c602b4f99af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:32.359333 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d23311cfc0484c4dbd23c602b4f99af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:32.361917 1438 tablet_bootstrap.cc:492] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901: Bootstrap starting.
I20260928 04:53:32.362563 1440 tablet_bootstrap.cc:492] T d23311cfc0484c4dbd23c602b4f99af0 P 3dbbaa2ccd844213819b12009a9a2123: Bootstrap starting.
I20260928 04:53:32.362754 1438 tablet_bootstrap.cc:654] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:32.363312 1440 tablet_bootstrap.cc:654] T d23311cfc0484c4dbd23c602b4f99af0 P 3dbbaa2ccd844213819b12009a9a2123: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:32.363369 1438 log.cc:824] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:32.363790 1439 tablet_bootstrap.cc:492] T d23311cfc0484c4dbd23c602b4f99af0 P e92358262c0145c281b2b75d584f536c: Bootstrap starting.
I20260928 04:53:32.363914 1440 log.cc:824] T d23311cfc0484c4dbd23c602b4f99af0 P 3dbbaa2ccd844213819b12009a9a2123: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:32.364293 1438 tablet_bootstrap.cc:492] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901: No bootstrap required, opened a new log
I20260928 04:53:32.364403 1438 ts_tablet_manager.cc:1397] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 04:53:32.364473 1439 tablet_bootstrap.cc:654] T d23311cfc0484c4dbd23c602b4f99af0 P e92358262c0145c281b2b75d584f536c: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:32.364705 1440 tablet_bootstrap.cc:492] T d23311cfc0484c4dbd23c602b4f99af0 P 3dbbaa2ccd844213819b12009a9a2123: No bootstrap required, opened a new log
I20260928 04:53:32.364787 1440 ts_tablet_manager.cc:1397] T d23311cfc0484c4dbd23c602b4f99af0 P 3dbbaa2ccd844213819b12009a9a2123: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:53:32.365043 1439 log.cc:824] T d23311cfc0484c4dbd23c602b4f99af0 P e92358262c0145c281b2b75d584f536c: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:32.366046 1439 tablet_bootstrap.cc:492] T d23311cfc0484c4dbd23c602b4f99af0 P e92358262c0145c281b2b75d584f536c: No bootstrap required, opened a new log
I20260928 04:53:32.366125 1439 ts_tablet_manager.cc:1397] T d23311cfc0484c4dbd23c602b4f99af0 P e92358262c0145c281b2b75d584f536c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:53:32.366534 1438 raft_consensus.cc:348] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c4288b7e734dd692c446fe005f2901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41059 } } peers { permanent_uuid: "3dbbaa2ccd844213819b12009a9a2123" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "e92358262c0145c281b2b75d584f536c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } }
I20260928 04:53:32.366739 1438 raft_consensus.cc:374] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:32.366773 1438 raft_consensus.cc:729] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58c4288b7e734dd692c446fe005f2901, State: Initialized, Role: FOLLOWER
I20260928 04:53:32.366680 1440 raft_consensus.cc:348] T d23311cfc0484c4dbd23c602b4f99af0 P 3dbbaa2ccd844213819b12009a9a2123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c4288b7e734dd692c446fe005f2901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41059 } } peers { permanent_uuid: "3dbbaa2ccd844213819b12009a9a2123" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "e92358262c0145c281b2b75d584f536c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } }
I20260928 04:53:32.366899 1440 raft_consensus.cc:374] T d23311cfc0484c4dbd23c602b4f99af0 P 3dbbaa2ccd844213819b12009a9a2123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:32.366896 1438 consensus_queue.cc:260] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c4288b7e734dd692c446fe005f2901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41059 } } peers { permanent_uuid: "3dbbaa2ccd844213819b12009a9a2123" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "e92358262c0145c281b2b75d584f536c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } }
I20260928 04:53:32.366950 1440 raft_consensus.cc:729] T d23311cfc0484c4dbd23c602b4f99af0 P 3dbbaa2ccd844213819b12009a9a2123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dbbaa2ccd844213819b12009a9a2123, State: Initialized, Role: FOLLOWER
I20260928 04:53:32.367089 1440 consensus_queue.cc:260] T d23311cfc0484c4dbd23c602b4f99af0 P 3dbbaa2ccd844213819b12009a9a2123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c4288b7e734dd692c446fe005f2901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41059 } } peers { permanent_uuid: "3dbbaa2ccd844213819b12009a9a2123" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "e92358262c0145c281b2b75d584f536c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } }
I20260928 04:53:32.367177 1438 ts_tablet_manager.cc:1428] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:53:32.367340 1287 heartbeater.cc:503] Master 127.0.98.62:38083 was elected leader, sending a full tablet report...
I20260928 04:53:32.367843 1440 ts_tablet_manager.cc:1428] T d23311cfc0484c4dbd23c602b4f99af0 P 3dbbaa2ccd844213819b12009a9a2123: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:53:32.367862 1421 heartbeater.cc:503] Master 127.0.98.62:38083 was elected leader, sending a full tablet report...
I20260928 04:53:32.368250 1439 raft_consensus.cc:348] T d23311cfc0484c4dbd23c602b4f99af0 P e92358262c0145c281b2b75d584f536c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c4288b7e734dd692c446fe005f2901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41059 } } peers { permanent_uuid: "3dbbaa2ccd844213819b12009a9a2123" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "e92358262c0145c281b2b75d584f536c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } }
I20260928 04:53:32.368459 1439 raft_consensus.cc:374] T d23311cfc0484c4dbd23c602b4f99af0 P e92358262c0145c281b2b75d584f536c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:32.368510 1439 raft_consensus.cc:729] T d23311cfc0484c4dbd23c602b4f99af0 P e92358262c0145c281b2b75d584f536c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92358262c0145c281b2b75d584f536c, State: Initialized, Role: FOLLOWER
I20260928 04:53:32.368646 1439 consensus_queue.cc:260] T d23311cfc0484c4dbd23c602b4f99af0 P e92358262c0145c281b2b75d584f536c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c4288b7e734dd692c446fe005f2901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41059 } } peers { permanent_uuid: "3dbbaa2ccd844213819b12009a9a2123" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "e92358262c0145c281b2b75d584f536c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } }
I20260928 04:53:32.368938 1439 ts_tablet_manager.cc:1428] T d23311cfc0484c4dbd23c602b4f99af0 P e92358262c0145c281b2b75d584f536c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:53:32.368963 1153 heartbeater.cc:503] Master 127.0.98.62:38083 was elected leader, sending a full tablet report...
I20260928 04:53:32.378276 1444 raft_consensus.cc:482] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:53:32.378473 1444 raft_consensus.cc:504] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c4288b7e734dd692c446fe005f2901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41059 } } peers { permanent_uuid: "3dbbaa2ccd844213819b12009a9a2123" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "e92358262c0145c281b2b75d584f536c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } }
I20260928 04:53:32.378974 1444 leader_election.cc:288] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3dbbaa2ccd844213819b12009a9a2123 (127.0.98.3:36803), e92358262c0145c281b2b75d584f536c (127.0.98.1:46103)
I20260928 04:53:32.381852 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:32.379161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46103 (local address 127.0.98.2:33309)
0928 04:53:32.379233 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:32.379246 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:32.379291 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:32.379450 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:32.379452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:32.379453 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:32.379523 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:32.379526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.380172 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.380173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:32.380946 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:32.380948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.381571 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.381571 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:32.381654 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:32.381657 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:53:32.381729 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 04:53:32.381775 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 04:53:32.381852 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:32.379254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33309 (local address 127.0.98.1:46103)
0928 04:53:32.379318 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:32.379320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:32.379331 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:53:32.379359 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:32.379360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:32.379372 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:32.379435 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:32.379599 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.380083 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.381055 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.381471 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.381569 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:32.381581 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:32.381739 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 04:53:32.381779 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 04:53:32.382014 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:32.379280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37761 (local address 127.0.98.3:36803)
0928 04:53:32.379325 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:32.379327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:32.379503 (+ 176us) server_negotiation.cc:407] Connection header received
0928 04:53:32.379564 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:32.379566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:32.379578 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:32.379636 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:32.379779 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.380263 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.381240 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.381639 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.381721 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:32.381732 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:32.381897 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 04:53:32.381941 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:53:32.382021 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:32.379183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36803 (local address 127.0.98.2:37761)
0928 04:53:32.379409 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:32.379417 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:32.379511 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:32.379641 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:32.379641 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:32.379643 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:32.379714 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:32.379715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.380363 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.380364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:32.381142 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:32.381144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.381734 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.381736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:32.381811 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:32.381815 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:53:32.381887 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 04:53:32.381929 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":68,"thread_start_us":113,"threads_started":1}
I20260928 04:53:32.382251 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d23311cfc0484c4dbd23c602b4f99af0" candidate_uuid: "58c4288b7e734dd692c446fe005f2901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92358262c0145c281b2b75d584f536c" is_pre_election: true
I20260928 04:53:32.382426 1107 raft_consensus.cc:2454] T d23311cfc0484c4dbd23c602b4f99af0 P e92358262c0145c281b2b75d584f536c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58c4288b7e734dd692c446fe005f2901 in term 0.
I20260928 04:53:32.382766 1176 leader_election.cc:302] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58c4288b7e734dd692c446fe005f2901, e92358262c0145c281b2b75d584f536c; no voters:
I20260928 04:53:32.382823 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d23311cfc0484c4dbd23c602b4f99af0" candidate_uuid: "58c4288b7e734dd692c446fe005f2901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dbbaa2ccd844213819b12009a9a2123" is_pre_election: true
I20260928 04:53:32.382965 1375 raft_consensus.cc:2454] T d23311cfc0484c4dbd23c602b4f99af0 P 3dbbaa2ccd844213819b12009a9a2123 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58c4288b7e734dd692c446fe005f2901 in term 0.
I20260928 04:53:32.382936 1444 raft_consensus.cc:2781] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:53:32.383015 1444 raft_consensus.cc:482] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:53:32.383040 1444 raft_consensus.cc:3037] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:32.383842 1444 raft_consensus.cc:504] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c4288b7e734dd692c446fe005f2901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41059 } } peers { permanent_uuid: "3dbbaa2ccd844213819b12009a9a2123" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "e92358262c0145c281b2b75d584f536c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } }
I20260928 04:53:32.384012 1444 leader_election.cc:288] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901 [CANDIDATE]: Term 1 election: Requested vote from peers 3dbbaa2ccd844213819b12009a9a2123 (127.0.98.3:36803), e92358262c0145c281b2b75d584f536c (127.0.98.1:46103)
I20260928 04:53:32.384215 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d23311cfc0484c4dbd23c602b4f99af0" candidate_uuid: "58c4288b7e734dd692c446fe005f2901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92358262c0145c281b2b75d584f536c"
I20260928 04:53:32.384255 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d23311cfc0484c4dbd23c602b4f99af0" candidate_uuid: "58c4288b7e734dd692c446fe005f2901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dbbaa2ccd844213819b12009a9a2123"
I20260928 04:53:32.384331 1375 raft_consensus.cc:3037] T d23311cfc0484c4dbd23c602b4f99af0 P 3dbbaa2ccd844213819b12009a9a2123 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:32.384323 1107 raft_consensus.cc:3037] T d23311cfc0484c4dbd23c602b4f99af0 P e92358262c0145c281b2b75d584f536c [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:32.385413 1107 raft_consensus.cc:2454] T d23311cfc0484c4dbd23c602b4f99af0 P e92358262c0145c281b2b75d584f536c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58c4288b7e734dd692c446fe005f2901 in term 1.
I20260928 04:53:32.385413 1375 raft_consensus.cc:2454] T d23311cfc0484c4dbd23c602b4f99af0 P 3dbbaa2ccd844213819b12009a9a2123 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58c4288b7e734dd692c446fe005f2901 in term 1.
I20260928 04:53:32.385604 1176 leader_election.cc:302] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58c4288b7e734dd692c446fe005f2901, e92358262c0145c281b2b75d584f536c; no voters:
I20260928 04:53:32.385725 1444 raft_consensus.cc:2781] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:32.385900 1444 raft_consensus.cc:686] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901 [term 1 LEADER]: Becoming Leader. State: Replica: 58c4288b7e734dd692c446fe005f2901, State: Running, Role: LEADER
I20260928 04:53:32.385982 1444 consensus_queue.cc:237] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c4288b7e734dd692c446fe005f2901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41059 } } peers { permanent_uuid: "3dbbaa2ccd844213819b12009a9a2123" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "e92358262c0145c281b2b75d584f536c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } }
I20260928 04:53:32.386864 961 catalog_manager.cc:5495] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58c4288b7e734dd692c446fe005f2901 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "58c4288b7e734dd692c446fe005f2901" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c4288b7e734dd692c446fe005f2901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dbbaa2ccd844213819b12009a9a2123" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e92358262c0145c281b2b75d584f536c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } health_report { overall_health: UNKNOWN } } }
Sep 28 04:53:32 dist-test-slave-m39l krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
W20260928 04:53:32.399714 1288 tablet.cc:2369] T d23311cfc0484c4dbd23c602b4f99af0 P 58c4288b7e734dd692c446fe005f2901: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:53:32.402890 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:32.394368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.0.98.1:46103)
0928 04:53:32.394452 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:32.394457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:32.394478 (+ 21us) server_negotiation.cc:407] Connection header received
0928 04:53:32.394621 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:32.394622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:32.394635 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:32.394680 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:32.395275 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.395778 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.396548 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.396736 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.396801 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:32.399456 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:32.399476 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:32.399476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:32.399486 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:32.401481 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:32.401860 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:32.401865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:32.401865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:32.401988 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:32.402219 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:32.402220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:32.402220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:32.402386 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:32.402609 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:32.402729 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 04:53:32.402813 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 04:53:32 dist-test-slave-m39l krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571212, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 04:53:32 dist-test-slave-m39l krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571212, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:53:32.429270 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:32.420529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59956 (local address 127.0.98.1:46103)
0928 04:53:32.420608 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:32.420610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:32.420673 (+ 63us) server_negotiation.cc:407] Connection header received
0928 04:53:32.420741 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:32.420749 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:32.420759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:32.420813 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:32.421519 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.422033 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.422949 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.423194 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.423251 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:32.426101 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:32.426103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:32.426103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:32.426111 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:32.427848 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:32.428276 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:32.428277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:32.428277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:32.428356 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:32.428557 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:32.428558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:32.428558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:32.428685 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:32.428781 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:32.429019 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 04:53:32.429101 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 04:53:32 dist-test-slave-m39l krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 04:53:32 dist-test-slave-m39l krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:53:32.454152 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:32.446121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59972 (local address 127.0.98.1:46103)
0928 04:53:32.446189 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:32.446191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:32.446232 (+ 41us) server_negotiation.cc:407] Connection header received
0928 04:53:32.446302 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:32.446303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:32.446312 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:32.446374 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:32.447121 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.447621 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.448512 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:32.448706 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:32.448772 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:32.451476 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:32.451478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:32.451478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:32.451487 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:32.452948 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:32.453345 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:32.453346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:32.453346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:32.453410 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:32.453673 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:32.453674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:32.453674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:32.453779 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:32.453853 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:32.453966 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 04:53:32.454041 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 04:53:32.454754 1067 tablet_service.cc:2908] Rejecting scan request for tablet d23311cfc0484c4dbd23c602b4f99af0: Uninitialized: clean time has not yet been initialized
I20260928 04:53:32.455776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 1024
I20260928 04:53:32.461421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 1158
I20260928 04:53:32.466405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 1292
I20260928 04:53:32.471100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 930
2026-09-28T04:53:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2090 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:53:32 dist-test-slave-m39l krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:53:32 dist-test-slave-m39l krb5kdc[1462](info): set up 2 sockets
Sep 28 04:53:32 dist-test-slave-m39l krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260928 04:53:33.514209 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:53:33 dist-test-slave-m39l krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:53:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:53:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:53:33.642599 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46551
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:33.730854 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:33.731161 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:33.731204 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:33.732585 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:53:33.732695 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 04:53:33.732719 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:33.732741 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:33.732775 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:53:33.732805 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:53:33.734686 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46551
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:33.735025 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:33.735291 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:33.740818 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:33.740835 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:33.741190 1475 server_base.cc:1034] running on GCE node
W20260928 04:53:33.740818 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:33.741568 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:33.741906 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:33.743072 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790571213743053 us; error 39 us; skew 500 ppm
Sep 28 04:53:33 dist-test-slave-m39l krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:33.745868 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:53:33.746409 1475 webserver.cc:460] Webserver started at http://127.0.98.62:46073/ using document root <none> and password file <none>
I20260928 04:53:33.746701 1475 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:33.746754 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:33.746856 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:33.748046 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "e5e9b63ad7374f9f89d756d23c504af5"
format_stamp: "Formatted at 2026-09-28 04:53:33 on dist-test-slave-m39l"
server_key: "43897105f8d4d577ee590e5eef2562e1"
server_key_iv: "db217ed20e1cd3808cd5fc35b93740be"
server_key_version: "encryptionkey@0"
I20260928 04:53:33.748377 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "e5e9b63ad7374f9f89d756d23c504af5"
format_stamp: "Formatted at 2026-09-28 04:53:33 on dist-test-slave-m39l"
server_key: "43897105f8d4d577ee590e5eef2562e1"
server_key_iv: "db217ed20e1cd3808cd5fc35b93740be"
server_key_version: "encryptionkey@0"
I20260928 04:53:33.749763 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:53:33.750607 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:33.750782 1475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:33.750872 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "e5e9b63ad7374f9f89d756d23c504af5"
format_stamp: "Formatted at 2026-09-28 04:53:33 on dist-test-slave-m39l"
server_key: "43897105f8d4d577ee590e5eef2562e1"
server_key_iv: "db217ed20e1cd3808cd5fc35b93740be"
server_key_version: "encryptionkey@0"
I20260928 04:53:33.750931 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:33.798458 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:33.801168 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:33.801296 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:33.806041 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46551
I20260928 04:53:33.806138 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46551 every 8 connection(s)
I20260928 04:53:33.806478 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:53:33.807287 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:33.808804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 1475
I20260928 04:53:33.808915 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:53:33.809085 392 external_mini_cluster.cc:1428] Setting key 69a35b2fd2feff5dc4732474c50f48cb
I20260928 04:53:33.810189 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5: Bootstrap starting.
I20260928 04:53:33.810949 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:33.811333 1553 log.cc:824] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:33.812105 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5: No bootstrap required, opened a new log
I20260928 04:53:33.814481 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e9b63ad7374f9f89d756d23c504af5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46551 } }
I20260928 04:53:33.814729 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:33.814777 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e9b63ad7374f9f89d756d23c504af5, State: Initialized, Role: FOLLOWER
I20260928 04:53:33.814908 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e9b63ad7374f9f89d756d23c504af5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46551 } }
I20260928 04:53:33.814985 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:53:33.815037 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:53:33.815088 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:33.815681 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e9b63ad7374f9f89d756d23c504af5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46551 } }
I20260928 04:53:33.815801 1553 leader_election.cc:302] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5e9b63ad7374f9f89d756d23c504af5; no voters:
I20260928 04:53:33.816011 1553 leader_election.cc:288] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:53:33.816080 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:33.816274 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5 [term 1 LEADER]: Becoming Leader. State: Replica: e5e9b63ad7374f9f89d756d23c504af5, State: Running, Role: LEADER
I20260928 04:53:33.816294 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:33.816406 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e9b63ad7374f9f89d756d23c504af5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46551 } }
Sep 28 04:53:33 dist-test-slave-m39l krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:33.818082 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5e9b63ad7374f9f89d756d23c504af5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e9b63ad7374f9f89d756d23c504af5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46551 } } }
I20260928 04:53:33.818185 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:33.818310 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5e9b63ad7374f9f89d756d23c504af5. Latest consensus state: current_term: 1 leader_uuid: "e5e9b63ad7374f9f89d756d23c504af5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e9b63ad7374f9f89d756d23c504af5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46551 } } }
I20260928 04:53:33.818385 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5 [sys.catalog]: This master's current role is: LEADER
W20260928 04:53:33.818583 1572 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:53:33.818643 1572 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:53:33.818701 1574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:53:33.818871 1574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:53:33.821060 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:33.809671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48412 (local address 127.0.98.62:46551)
0928 04:53:33.809834 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:33.809848 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:33.809875 (+ 27us) server_negotiation.cc:407] Connection header received
0928 04:53:33.810359 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:33.810368 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:33.810422 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:33.810601 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:33.811505 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:33.812493 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:33.813636 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:33.814085 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:33.814193 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:33.817128 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:33.817133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:33.817134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:33.817149 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:33.819422 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:33.819928 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:33.819931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:33.819931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:33.820028 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:33.820253 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:33.820254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:33.820254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:33.820500 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:33.820648 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:33.820781 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 04:53:33.820898 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260928 04:53:33.823853 1574 catalog_manager.cc:1347] Generated new cluster ID: 4f3b9a5a1e464040bceef8b817b94069
I20260928 04:53:33.823956 1574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:53:33.837520 1574 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:53:33.838296 1574 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:53:33.844511 1574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5: Generated new TSK 0
I20260928 04:53:33.844743 1574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:53:33.882871 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46551
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:33.968775 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:33.968994 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:33.969020 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:33.970142 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:33.970177 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:33.970197 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:53:33.970216 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:53:33.971756 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:33.972092 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:33.972328 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:53:33.977850 1579 server_base.cc:1034] running on GCE node
W20260928 04:53:33.977814 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:33.977790 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:33.977821 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:33.978243 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:33.978480 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:33.979729 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790571213979715 us; error 29 us; skew 500 ppm
Sep 28 04:53:33 dist-test-slave-m39l krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:33.982602 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:53:33.983103 1579 webserver.cc:460] Webserver started at http://127.0.98.1:45745/ using document root <none> and password file <none>
I20260928 04:53:33.983336 1579 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:33.983384 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:33.983518 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:33.984652 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "f8630f596a3f4706801f0995c2e80917"
format_stamp: "Formatted at 2026-09-28 04:53:33 on dist-test-slave-m39l"
server_key: "e99b8bc41a4895f697d4c088538522e2"
server_key_iv: "556b311f7f7508b5393423b62eb35126"
server_key_version: "encryptionkey@0"
I20260928 04:53:33.985023 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f8630f596a3f4706801f0995c2e80917"
format_stamp: "Formatted at 2026-09-28 04:53:33 on dist-test-slave-m39l"
server_key: "e99b8bc41a4895f697d4c088538522e2"
server_key_iv: "556b311f7f7508b5393423b62eb35126"
server_key_version: "encryptionkey@0"
I20260928 04:53:33.986397 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:53:33.987258 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:33.987495 1579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:33.987600 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "f8630f596a3f4706801f0995c2e80917"
format_stamp: "Formatted at 2026-09-28 04:53:33 on dist-test-slave-m39l"
server_key: "e99b8bc41a4895f697d4c088538522e2"
server_key_iv: "556b311f7f7508b5393423b62eb35126"
server_key_version: "encryptionkey@0"
I20260928 04:53:33.987676 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:33.995899 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:33.998679 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:33.998875 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:33.999435 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:33.999480 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:33.999504 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:33.999547 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 04:53:34 dist-test-slave-m39l krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:34.006734 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41025
I20260928 04:53:34.006817 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41025 every 8 connection(s)
I20260928 04:53:34.007160 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:53:34.007848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 1579
I20260928 04:53:34.007941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:53:34.008179 392 external_mini_cluster.cc:1428] Setting key c3b1a1ee3062bfdcbdfeeaa279af08c8
I20260928 04:53:34.010820 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.000049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42515 (local address 127.0.98.62:46551)
0928 04:53:34.000099 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:34.000101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:34.000319 (+ 218us) server_negotiation.cc:407] Connection header received
0928 04:53:34.000864 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:34.000866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:34.000881 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:34.000968 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:34.001890 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.002808 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.003889 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.004251 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.004343 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.007307 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:34.007309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:34.007309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:34.007319 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:34.009241 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:34.009708 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:34.009710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:34.009711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:34.009778 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:34.010054 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:34.010056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:34.010056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:34.010184 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:34.010265 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:34.010617 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 04:53:34.010715 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:53:34.011031 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.000029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46551 (local address 127.0.98.1:42515)
0928 04:53:34.000200 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:34.000238 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:34.000761 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:34.001028 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:34.001033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:34.001411 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:34.001781 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.001787 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.002913 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.002914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.003819 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.003821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.004390 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.004391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.004526 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.005119 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:34.005129 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:34.007188 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:34.009349 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:34.009358 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:34.009360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:34.009609 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:34.009858 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:34.009859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:34.009860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:34.009984 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:34.010256 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:34.010262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:34.010419 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 04:53:34.010595 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 04:53:34.010776 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260928 04:53:34.012567 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46551
I20260928 04:53:34.012665 1710 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:34.012960 1710 heartbeater.cc:511] Master 127.0.98.62:46551 requested a full tablet report, sending...
I20260928 04:53:34.013564 1506 ts_manager.cc:194] Registered new tserver with Master: f8630f596a3f4706801f0995c2e80917 (127.0.98.1:41025)
I20260928 04:53:34.014671 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42515
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 04:53:34.020251 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.012741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46551 (local address 127.0.98.1:35825)
0928 04:53:34.013017 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:34.013030 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:34.013140 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:34.013379 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:34.013380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:34.013622 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:34.013694 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.013695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.014717 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.014718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.015737 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.015741 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.016126 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.016127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.016203 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.016732 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:34.016737 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:34.017099 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:34.019221 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:34.019222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:34.019222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:34.019405 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:34.019645 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:34.019645 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:34.019646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:34.019698 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:34.019965 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:34.019966 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:34.020008 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 04:53:34.020081 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 04:53:34.020168 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":81,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 04:53:34.020457 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.012760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35825 (local address 127.0.98.62:46551)
0928 04:53:34.012806 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:34.012808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:34.013112 (+ 304us) server_negotiation.cc:407] Connection header received
0928 04:53:34.013213 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:34.013214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:34.013227 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:34.013296 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:34.013765 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.014620 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.015834 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.016043 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.016110 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.017193 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:34.017195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:34.017196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:34.017202 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:34.019132 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:34.019474 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:34.019475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:34.019476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:34.019528 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:34.019770 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:34.019771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:34.019771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:34.019903 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:34.019969 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:34.020166 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 04:53:34.020297 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:53:34.038005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46551
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:34.121294 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:34.121536 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:34.121562 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:34.122845 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:34.122890 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:34.122911 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:53:34.122929 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:53:34.124549 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:34.124755 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:34.124945 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:34.130121 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:34.130230 1719 server_base.cc:1034] running on GCE node
W20260928 04:53:34.130133 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:34.130182 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:34.130672 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:34.130977 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:34.132169 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790571214132163 us; error 35 us; skew 500 ppm
Sep 28 04:53:34 dist-test-slave-m39l krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:34.134865 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:53:34.135290 1719 webserver.cc:460] Webserver started at http://127.0.98.2:36689/ using document root <none> and password file <none>
I20260928 04:53:34.135483 1719 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:34.135526 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:34.135617 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:34.136654 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1"
format_stamp: "Formatted at 2026-09-28 04:53:34 on dist-test-slave-m39l"
server_key: "33398ab5f439d96b597c9bf1b470d268"
server_key_iv: "2e82a685f584e9c4b11bce91c1ce4ee5"
server_key_version: "encryptionkey@0"
I20260928 04:53:34.136967 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1"
format_stamp: "Formatted at 2026-09-28 04:53:34 on dist-test-slave-m39l"
server_key: "33398ab5f439d96b597c9bf1b470d268"
server_key_iv: "2e82a685f584e9c4b11bce91c1ce4ee5"
server_key_version: "encryptionkey@0"
I20260928 04:53:34.138306 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:53:34.139168 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:34.139331 1719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:53:34.139420 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1"
format_stamp: "Formatted at 2026-09-28 04:53:34 on dist-test-slave-m39l"
server_key: "33398ab5f439d96b597c9bf1b470d268"
server_key_iv: "2e82a685f584e9c4b11bce91c1ce4ee5"
server_key_version: "encryptionkey@0"
I20260928 04:53:34.139477 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:34.152483 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:34.154951 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:34.155077 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:34.155534 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:34.155570 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:34.155592 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:34.155648 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:34.162791 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41413
I20260928 04:53:34.162882 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41413 every 8 connection(s)
Sep 28 04:53:34 dist-test-slave-m39l krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:34.163219 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:53:34.166419 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.156174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50317 (local address 127.0.98.62:46551)
0928 04:53:34.156222 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:34.156229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:34.156461 (+ 232us) server_negotiation.cc:407] Connection header received
0928 04:53:34.157033 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:34.157034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:34.157047 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:34.157130 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:34.158130 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.159017 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.160627 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.160851 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.160945 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.163800 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:34.163802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:34.163802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:34.163811 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:34.165030 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:34.165452 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:34.165453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:34.165454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:34.165514 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:34.165760 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:34.165761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:34.165761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:34.165886 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:34.165962 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:34.166226 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 04:53:34.166309 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:53:34.166643 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.156143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46551 (local address 127.0.98.2:50317)
0928 04:53:34.156322 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:34.156372 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:34.156922 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:34.157248 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:34.157254 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:34.157678 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:34.157997 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.158004 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.159116 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.159117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.160507 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.160511 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.160956 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.160957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.161101 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.161725 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:34.161737 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:34.163689 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:34.165128 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:34.165135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:34.165137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:34.165356 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:34.165579 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:34.165580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:34.165580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:34.165693 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:34.165951 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:34.165957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:34.166099 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 04:53:34.166208 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 04:53:34.166375 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260928 04:53:34.167150 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46551
I20260928 04:53:34.167230 1850 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:34.167428 1850 heartbeater.cc:511] Master 127.0.98.62:46551 requested a full tablet report, sending...
I20260928 04:53:34.167821 1506 ts_manager.cc:194] Registered new tserver with Master: 2bef0189d9fe4ae7baed7b3e2267c9e1 (127.0.98.2:41413)
I20260928 04:53:34.168325 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50317
I20260928 04:53:34.173297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 1719
I20260928 04:53:34.173406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:53:34.173652 392 external_mini_cluster.cc:1428] Setting key 1913a09fde13f3417356b1db9e5af842
I20260928 04:53:34.174347 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.168270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47479 (local address 127.0.98.62:46551)
0928 04:53:34.168312 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:34.168314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:34.168392 (+ 78us) server_negotiation.cc:407] Connection header received
0928 04:53:34.168486 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:34.168487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:34.168497 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:34.168545 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:34.168901 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.169377 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.170145 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.170324 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.170379 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.171366 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:34.171367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:34.171367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:34.171373 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:34.172840 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:34.173197 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:34.173198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:34.173199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:34.173277 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:34.173559 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:34.173560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:34.173561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:34.173803 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:34.173882 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:34.174150 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 04:53:34.174248 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:53:34.175748 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.168190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46551 (local address 127.0.98.2:47479)
0928 04:53:34.168322 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:34.168334 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:34.168400 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:34.168547 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:34.168548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:34.168760 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:34.168825 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.168827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.169493 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.169494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.170063 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.170065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.170394 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.170395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.170447 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.170932 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:34.170936 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:34.171294 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:34.172917 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:34.172919 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:34.172919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:34.173102 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:34.173355 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:34.173356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:34.173356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:34.173447 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:34.173961 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:34.173962 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:34.174023 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 04:53:34.174121 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 04:53:34.174247 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:53:34.201694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46551
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:34.286935 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:34.287169 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:34.287196 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:34.288451 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:34.288502 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:34.288527 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:53:34.288546 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:53:34.290161 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:34.290412 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:34.290699 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:34.296499 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:34.296499 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:34.296533 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:34.296789 1859 server_base.cc:1034] running on GCE node
I20260928 04:53:34.297113 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:34.297379 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:34.298612 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790571214298589 us; error 36 us; skew 500 ppm
Sep 28 04:53:34 dist-test-slave-m39l krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:34.301438 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:53:34.302028 1859 webserver.cc:460] Webserver started at http://127.0.98.3:41183/ using document root <none> and password file <none>
I20260928 04:53:34.302297 1859 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:34.302367 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:34.302572 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:34.303753 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "63d521c1f2ed498aa1f533997a1b48e1"
format_stamp: "Formatted at 2026-09-28 04:53:34 on dist-test-slave-m39l"
server_key: "1564ed0688e47ce28dfb0020df1b2f35"
server_key_iv: "fe8c42ef6335ede84bbc74d59cedc5b4"
server_key_version: "encryptionkey@0"
I20260928 04:53:34.304165 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "63d521c1f2ed498aa1f533997a1b48e1"
format_stamp: "Formatted at 2026-09-28 04:53:34 on dist-test-slave-m39l"
server_key: "1564ed0688e47ce28dfb0020df1b2f35"
server_key_iv: "fe8c42ef6335ede84bbc74d59cedc5b4"
server_key_version: "encryptionkey@0"
I20260928 04:53:34.305680 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:53:34.306700 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:34.306895 1859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:34.307001 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "63d521c1f2ed498aa1f533997a1b48e1"
format_stamp: "Formatted at 2026-09-28 04:53:34 on dist-test-slave-m39l"
server_key: "1564ed0688e47ce28dfb0020df1b2f35"
server_key_iv: "fe8c42ef6335ede84bbc74d59cedc5b4"
server_key_version: "encryptionkey@0"
I20260928 04:53:34.307083 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:34.328258 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:34.330572 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:34.330736 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:34.331399 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:34.331467 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:34.331522 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:34.331537 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 04:53:34 dist-test-slave-m39l krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:34.338915 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32943
I20260928 04:53:34.339008 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32943 every 8 connection(s)
I20260928 04:53:34.339288 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:53:34.341727 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.331887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41185 (local address 127.0.98.62:46551)
0928 04:53:34.331957 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:34.331959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:34.332219 (+ 260us) server_negotiation.cc:407] Connection header received
0928 04:53:34.333000 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:34.333002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:34.333015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:34.333105 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:34.334071 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.334850 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.335974 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.336182 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.336267 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.338636 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:34.338638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:34.338638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:34.338646 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:34.340243 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:34.340660 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:34.340661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:34.340662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:34.340715 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:34.341041 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:34.341042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:34.341042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:34.341154 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:34.341210 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:34.341579 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 04:53:34.341657 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:53:34.342010 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.331874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46551 (local address 127.0.98.3:41185)
0928 04:53:34.332050 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:34.332107 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:34.332859 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:34.333189 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:34.333195 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:34.333608 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:34.333946 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.333954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.334968 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.334968 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.335891 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.335893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.336292 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.336293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.336419 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.336980 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:34.336992 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:34.338537 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:34.340325 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:34.340331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:34.340332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:34.340587 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:34.340813 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:34.340814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:34.340814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:34.340964 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:34.341240 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:34.341246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:34.341402 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 04:53:34.341513 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 04:53:34.341655 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260928 04:53:34.342599 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46551
I20260928 04:53:34.342687 1990 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:34.342945 1990 heartbeater.cc:511] Master 127.0.98.62:46551 requested a full tablet report, sending...
I20260928 04:53:34.343360 1506 ts_manager.cc:194] Registered new tserver with Master: 63d521c1f2ed498aa1f533997a1b48e1 (127.0.98.3:32943)
I20260928 04:53:34.343851 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41185
I20260928 04:53:34.347316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 1859
I20260928 04:53:34.347427 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:53:34.347623 392 external_mini_cluster.cc:1428] Setting key 3f4ec72ca2ce56c8a7d12a0af531051f
I20260928 04:53:34.348863 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:53:34.350360 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.343757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46551 (local address 127.0.98.3:54465)
0928 04:53:34.343880 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:34.343892 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:34.343962 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:34.344077 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:34.344077 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:34.344271 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:34.344359 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.344361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.345321 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.345323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.345837 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.345839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.346214 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.346215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.346265 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.346763 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:34.346767 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:34.347109 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:34.349304 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:34.349305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:34.349306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:34.349504 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:34.349746 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:34.349747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:34.349747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:34.349805 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:34.350083 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:34.350083 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:34.350123 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 04:53:34.350184 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 04:53:34.350276 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
I20260928 04:53:34.350477 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.343825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54465 (local address 127.0.98.62:46551)
0928 04:53:34.343983 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:34.343985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:34.343993 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:53:34.344017 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:34.344018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:34.344026 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:34.344070 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:34.344515 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.345212 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.345925 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.346139 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.346199 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.347235 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:34.347237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:34.347238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:34.347246 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:34.349204 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:34.349584 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:34.349585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:34.349586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:34.349649 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:34.349866 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:34.349867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:34.349868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:34.350017 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:34.350085 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:34.350223 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 04:53:34.350305 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
Sep 28 04:53:34 dist-test-slave-m39l krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571214, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 04:53:34 dist-test-slave-m39l krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571214, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:34.383980 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.374808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.0.98.62:46551)
0928 04:53:34.374901 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:34.374904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:34.374916 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:53:34.374948 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:34.374949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:34.374962 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:34.375065 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:34.375759 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.376273 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.377561 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.377764 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.377837 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.380984 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:34.380986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:34.380987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:34.381004 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:34.382459 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:34.383058 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:34.383059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:34.383059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:34.383124 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:34.383309 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:34.383310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:34.383310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:34.383443 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:34.383530 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:34.383648 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 04:53:34.383740 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:53:34.386190 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:53:34.388198 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:53:34.395216 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.390873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41025 (local address 127.0.0.1:51486)
0928 04:53:34.391102 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:34.391185 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:34.391263 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:34.391592 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:34.391663 (+ 71us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:34.391665 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:34.391695 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.391697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.393115 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.393116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.394035 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.394038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.394874 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.394875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.394959 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.394974 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 04:53:34.395057 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 04:53:34.395115 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":70,"thread_start_us":91,"threads_started":1}
I20260928 04:53:34.395241 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.390987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51486 (local address 127.0.98.1:41025)
0928 04:53:34.391381 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:34.391387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:34.391401 (+ 14us) server_negotiation.cc:407] Connection header received
0928 04:53:34.391448 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:34.391450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:34.391467 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:34.391568 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:34.391787 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.392974 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.394144 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.394794 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.394874 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.394965 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:34.395085 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 04:53:34.395128 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":135,"threads_started":1}
I20260928 04:53:34.395687 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.390911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32943 (local address 127.0.0.1:47984)
0928 04:53:34.391372 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:34.391381 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:34.391420 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:34.391906 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:34.391907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:34.391909 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:34.391967 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.391970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.392973 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.392974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.394324 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.394327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.395442 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.395443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.395515 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.395520 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:53:34.395566 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 04:53:34.395617 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":51,"thread_start_us":96,"threads_started":1}
I20260928 04:53:34.396071 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.391260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47984 (local address 127.0.98.3:32943)
0928 04:53:34.391697 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:34.391702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:34.391712 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:53:34.391749 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:34.391751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:34.391765 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:34.391838 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:34.392034 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.392868 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.394421 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.395205 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.395268 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.395905 (+ 637us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:34.395934 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 04:53:34.395976 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":70,"threads_started":1}
I20260928 04:53:34.396114 1644 tablet_service.cc:1467] Processing CreateTablet for tablet aea41e78d38f496586819440ad0a2b35 (DEFAULT_TABLE table=test-table [id=af8c0f61ff7442d3b64ea8669dc4d493]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:34.396437 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.390895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41413 (local address 127.0.0.1:52312)
0928 04:53:34.391259 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:34.391268 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:34.391310 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:34.391596 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:34.391598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:34.391600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:34.391679 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.391682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.393095 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.393096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.394699 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.394702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.396113 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.396114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.396210 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.396214 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:53:34.396261 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 04:53:34.396341 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":73,"thread_start_us":88,"threads_started":1}
I20260928 04:53:34.396517 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aea41e78d38f496586819440ad0a2b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:34.396700 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.391180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52312 (local address 127.0.98.2:41413)
0928 04:53:34.391438 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:34.391442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:34.391451 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:53:34.391485 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:34.391487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:34.391503 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:34.391582 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:34.391773 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.392997 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.394799 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.395926 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.396006 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.396082 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:34.396542 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 04:53:34.396589 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":76,"threads_started":1}
I20260928 04:53:34.396735 1924 tablet_service.cc:1467] Processing CreateTablet for tablet aea41e78d38f496586819440ad0a2b35 (DEFAULT_TABLE table=test-table [id=af8c0f61ff7442d3b64ea8669dc4d493]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:34.397037 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aea41e78d38f496586819440ad0a2b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:34.400046 2013 tablet_bootstrap.cc:492] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1: Bootstrap starting.
I20260928 04:53:34.400677 2014 tablet_bootstrap.cc:492] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917: Bootstrap starting.
I20260928 04:53:34.400931 2013 tablet_bootstrap.cc:654] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:34.400810 1784 tablet_service.cc:1467] Processing CreateTablet for tablet aea41e78d38f496586819440ad0a2b35 (DEFAULT_TABLE table=test-table [id=af8c0f61ff7442d3b64ea8669dc4d493]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:34.401221 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aea41e78d38f496586819440ad0a2b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:34.401292 2013 log.cc:824] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:34.401383 2014 tablet_bootstrap.cc:654] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:34.401717 2014 log.cc:824] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:34.402371 2013 tablet_bootstrap.cc:492] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1: No bootstrap required, opened a new log
I20260928 04:53:34.402442 2014 tablet_bootstrap.cc:492] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917: No bootstrap required, opened a new log
I20260928 04:53:34.402462 2013 ts_tablet_manager.cc:1397] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:53:34.402515 2014 ts_tablet_manager.cc:1397] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:53:34.404560 2013 raft_consensus.cc:348] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } }
I20260928 04:53:34.404560 2014 raft_consensus.cc:348] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } }
I20260928 04:53:34.404744 2014 raft_consensus.cc:374] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:34.404744 2013 raft_consensus.cc:374] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:34.404788 2014 raft_consensus.cc:729] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8630f596a3f4706801f0995c2e80917, State: Initialized, Role: FOLLOWER
I20260928 04:53:34.404788 2013 raft_consensus.cc:729] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d521c1f2ed498aa1f533997a1b48e1, State: Initialized, Role: FOLLOWER
I20260928 04:53:34.404923 2013 consensus_queue.cc:260] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } }
I20260928 04:53:34.404923 2014 consensus_queue.cc:260] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } }
I20260928 04:53:34.405205 2014 ts_tablet_manager.cc:1428] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:53:34.405215 2017 tablet_bootstrap.cc:492] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1: Bootstrap starting.
I20260928 04:53:34.405376 1710 heartbeater.cc:503] Master 127.0.98.62:46551 was elected leader, sending a full tablet report...
I20260928 04:53:34.405416 2013 ts_tablet_manager.cc:1428] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:53:34.405699 1990 heartbeater.cc:503] Master 127.0.98.62:46551 was elected leader, sending a full tablet report...
I20260928 04:53:34.406183 2017 tablet_bootstrap.cc:654] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:34.406644 2017 log.cc:824] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:34.407589 2017 tablet_bootstrap.cc:492] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1: No bootstrap required, opened a new log
I20260928 04:53:34.407684 2017 ts_tablet_manager.cc:1397] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 04:53:34.409900 2017 raft_consensus.cc:348] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } }
I20260928 04:53:34.410126 2017 raft_consensus.cc:374] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:34.410175 2017 raft_consensus.cc:729] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bef0189d9fe4ae7baed7b3e2267c9e1, State: Initialized, Role: FOLLOWER
I20260928 04:53:34.410328 2017 consensus_queue.cc:260] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } }
I20260928 04:53:34.410717 2017 ts_tablet_manager.cc:1428] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:53:34.410765 1850 heartbeater.cc:503] Master 127.0.98.62:46551 was elected leader, sending a full tablet report...
W20260928 04:53:34.413892 1851 tablet.cc:2369] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:53:34.476832 2021 raft_consensus.cc:482] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:53:34.476987 2021 raft_consensus.cc:504] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } }
I20260928 04:53:34.477494 2021 leader_election.cc:288] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8630f596a3f4706801f0995c2e80917 (127.0.98.1:41025), 63d521c1f2ed498aa1f533997a1b48e1 (127.0.98.3:32943)
I20260928 04:53:34.480786 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.477913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48541 (local address 127.0.98.3:32943)
0928 04:53:34.477955 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:34.477957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:34.478087 (+ 130us) server_negotiation.cc:407] Connection header received
0928 04:53:34.478137 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:34.478138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:34.478152 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:34.478218 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:34.478417 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.478951 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.480015 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.480384 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.480474 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.480486 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:34.480658 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 04:53:34.480697 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:53:34.480798 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.477777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51265 (local address 127.0.98.1:41025)
0928 04:53:34.477911 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:34.477914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:34.477930 (+ 16us) server_negotiation.cc:407] Connection header received
0928 04:53:34.477973 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:34.477974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:34.477991 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:34.478084 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:34.478425 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.478943 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.480021 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.480411 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.480501 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.480512 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:34.480676 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 04:53:34.480720 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 04:53:34.480827 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.477729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41025 (local address 127.0.98.2:51265)
0928 04:53:34.477807 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:34.477819 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:34.477867 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:34.478257 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:34.478258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:34.478260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:34.478314 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.478316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.479052 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.479053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.479901 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.479903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.480515 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.480516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.480594 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.480598 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:53:34.480670 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 04:53:34.480720 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":44}
I20260928 04:53:34.480841 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.477838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32943 (local address 127.0.98.2:48541)
0928 04:53:34.478036 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:34.478043 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:34.478084 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:34.478233 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:34.478234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:34.478236 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:34.478302 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.478304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.479060 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.479061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.479898 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.479900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.480475 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.480477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.480566 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.480569 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:53:34.480640 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 04:53:34.480683 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":1,"thread_start_us":153,"threads_started":1}
I20260928 04:53:34.481276 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aea41e78d38f496586819440ad0a2b35" candidate_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8630f596a3f4706801f0995c2e80917" is_pre_election: true
I20260928 04:53:34.481279 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aea41e78d38f496586819440ad0a2b35" candidate_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d521c1f2ed498aa1f533997a1b48e1" is_pre_election: true
I20260928 04:53:34.481453 1664 raft_consensus.cc:2454] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bef0189d9fe4ae7baed7b3e2267c9e1 in term 0.
I20260928 04:53:34.481468 1944 raft_consensus.cc:2454] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bef0189d9fe4ae7baed7b3e2267c9e1 in term 0.
I20260928 04:53:34.481746 1738 leader_election.cc:302] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bef0189d9fe4ae7baed7b3e2267c9e1, 63d521c1f2ed498aa1f533997a1b48e1; no voters:
I20260928 04:53:34.482043 2021 raft_consensus.cc:2781] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:53:34.482128 2021 raft_consensus.cc:482] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:53:34.482167 2021 raft_consensus.cc:3037] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:34.484954 2021 raft_consensus.cc:504] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } }
I20260928 04:53:34.485201 2021 leader_election.cc:288] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [CANDIDATE]: Term 1 election: Requested vote from peers f8630f596a3f4706801f0995c2e80917 (127.0.98.1:41025), 63d521c1f2ed498aa1f533997a1b48e1 (127.0.98.3:32943)
I20260928 04:53:34.485376 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aea41e78d38f496586819440ad0a2b35" candidate_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8630f596a3f4706801f0995c2e80917"
I20260928 04:53:34.485477 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aea41e78d38f496586819440ad0a2b35" candidate_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d521c1f2ed498aa1f533997a1b48e1"
I20260928 04:53:34.485514 1664 raft_consensus.cc:3037] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:34.485558 1944 raft_consensus.cc:3037] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:34.486536 1944 raft_consensus.cc:2454] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bef0189d9fe4ae7baed7b3e2267c9e1 in term 1.
I20260928 04:53:34.486518 1664 raft_consensus.cc:2454] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bef0189d9fe4ae7baed7b3e2267c9e1 in term 1.
I20260928 04:53:34.486729 1735 leader_election.cc:302] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bef0189d9fe4ae7baed7b3e2267c9e1, f8630f596a3f4706801f0995c2e80917; no voters:
I20260928 04:53:34.486868 2021 raft_consensus.cc:2781] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:34.487057 2021 raft_consensus.cc:686] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 1 LEADER]: Becoming Leader. State: Replica: 2bef0189d9fe4ae7baed7b3e2267c9e1, State: Running, Role: LEADER
I20260928 04:53:34.487268 2021 consensus_queue.cc:237] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } }
I20260928 04:53:34.488379 1507 catalog_manager.cc:5495] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bef0189d9fe4ae7baed7b3e2267c9e1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } health_report { overall_health: HEALTHY } } }
W20260928 04:53:34.508010 1711 tablet.cc:2369] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:53:34.517897 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.514142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52322 (local address 127.0.98.2:41413)
0928 04:53:34.514191 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:34.514194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:34.514264 (+ 70us) server_negotiation.cc:407] Connection header received
0928 04:53:34.514339 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:34.514340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:34.514360 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:34.514466 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:53:34.514668 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.515411 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.516599 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.516834 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.516935 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.517283 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:53:34.517463 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:53:34.517630 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:34.517791 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 04:53:34.517824 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:53:34.520267 1664 raft_consensus.cc:1261] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917 [term 1 FOLLOWER]: Refusing update from remote peer 2bef0189d9fe4ae7baed7b3e2267c9e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:34.520267 1944 raft_consensus.cc:1261] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1 [term 1 FOLLOWER]: Refusing update from remote peer 2bef0189d9fe4ae7baed7b3e2267c9e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:34.520696 2021 consensus_queue.cc:1035] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:34.520874 2023 consensus_queue.cc:1035] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:34.523907 2026 mvcc.cc:205] Tried to move back new op lower bound from 7334179694672113664 to 7334179694540730368. Current Snapshot: MvccSnapshot[applied={T|T < 7334179694672113664}]
I20260928 04:53:34.525231 2028 mvcc.cc:205] Tried to move back new op lower bound from 7334179694672113664 to 7334179694540730368. Current Snapshot: MvccSnapshot[applied={T|T < 7334179694672113664 or (T in {7334179694672113664})}]
I20260928 04:53:34.525434 2027 mvcc.cc:205] Tried to move back new op lower bound from 7334179694672113664 to 7334179694540730368. Current Snapshot: MvccSnapshot[applied={T|T < 7334179694672113664}]
I20260928 04:53:34.528163 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:53:34.528261 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:53:34.529246 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5: Sending DeleteTablet for 3 replicas of tablet aea41e78d38f496586819440ad0a2b35
I20260928 04:53:34.529619 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet aea41e78d38f496586819440ad0a2b35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52312
I20260928 04:53:34.529609 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet aea41e78d38f496586819440ad0a2b35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51486
I20260928 04:53:34.529663 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet aea41e78d38f496586819440ad0a2b35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47984
I20260928 04:53:34.529821 2036 tablet_replica.cc:330] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917: stopping tablet replica
I20260928 04:53:34.529914 2038 tablet_replica.cc:330] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1: stopping tablet replica
I20260928 04:53:34.529934 2036 raft_consensus.cc:2229] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:53:34.530011 2038 raft_consensus.cc:2229] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:53:34.530045 2036 raft_consensus.cc:2258] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:34.530242 2037 tablet_replica.cc:330] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1: stopping tablet replica
I20260928 04:53:34.530359 2037 raft_consensus.cc:2229] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:53:34.530428 2038 raft_consensus.cc:2258] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:34.530458 2037 raft_consensus.cc:2258] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:34.530583 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:53:34.530761 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:53:34.531623 2038 ts_tablet_manager.cc:1905] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:34.531492 2036 ts_tablet_manager.cc:1905] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:34.532250 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 8af9fa78a98c42f5a68d17b34f104bd9 (DEFAULT_TABLE table=test-table [id=f1be29d73fbb4061901ef76d5afd8860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:34.532371 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af9fa78a98c42f5a68d17b34f104bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:34.533514 2017 tablet_bootstrap.cc:492] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1: Bootstrap starting.
I20260928 04:53:34.531032 2037 ts_tablet_manager.cc:1905] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:34.534017 2017 tablet_bootstrap.cc:654] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:34.534401 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 8af9fa78a98c42f5a68d17b34f104bd9 (DEFAULT_TABLE table=test-table [id=f1be29d73fbb4061901ef76d5afd8860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:34.534524 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 8af9fa78a98c42f5a68d17b34f104bd9 (DEFAULT_TABLE table=test-table [id=f1be29d73fbb4061901ef76d5afd8860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:34.534554 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af9fa78a98c42f5a68d17b34f104bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:34.535073 2017 tablet_bootstrap.cc:492] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1: No bootstrap required, opened a new log
I20260928 04:53:34.535136 2017 ts_tablet_manager.cc:1397] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:53:34.535667 2014 tablet_bootstrap.cc:492] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917: Bootstrap starting.
I20260928 04:53:34.535773 2017 raft_consensus.cc:348] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } }
I20260928 04:53:34.535857 2017 raft_consensus.cc:374] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:34.535882 2017 raft_consensus.cc:729] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bef0189d9fe4ae7baed7b3e2267c9e1, State: Initialized, Role: FOLLOWER
I20260928 04:53:34.535935 2017 consensus_queue.cc:260] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } }
I20260928 04:53:34.536059 2017 ts_tablet_manager.cc:1428] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:34.536064 2014 tablet_bootstrap.cc:654] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:34.537371 2014 tablet_bootstrap.cc:492] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917: No bootstrap required, opened a new log
I20260928 04:53:34.537415 2014 ts_tablet_manager.cc:1397] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:53:34.537597 2014 raft_consensus.cc:348] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } }
I20260928 04:53:34.537662 2014 raft_consensus.cc:374] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:34.537675 2014 raft_consensus.cc:729] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8630f596a3f4706801f0995c2e80917, State: Initialized, Role: FOLLOWER
I20260928 04:53:34.537731 2014 consensus_queue.cc:260] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } }
I20260928 04:53:34.537842 2014 ts_tablet_manager.cc:1428] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:34.597026 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af9fa78a98c42f5a68d17b34f104bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:34.598073 2038 ts_tablet_manager.cc:1918] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:53:34.598259 2038 log.cc:1196] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/aea41e78d38f496586819440ad0a2b35
I20260928 04:53:34.598666 2038 ts_tablet_manager.cc:1939] T aea41e78d38f496586819440ad0a2b35 P 63d521c1f2ed498aa1f533997a1b48e1: Deleting consensus metadata
I20260928 04:53:34.598775 2013 tablet_bootstrap.cc:492] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1: Bootstrap starting.
I20260928 04:53:34.599052 2037 ts_tablet_manager.cc:1918] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:53:34.599196 2037 log.cc:1196] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/aea41e78d38f496586819440ad0a2b35
I20260928 04:53:34.599244 2013 tablet_bootstrap.cc:654] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:34.599567 2037 ts_tablet_manager.cc:1939] T aea41e78d38f496586819440ad0a2b35 P 2bef0189d9fe4ae7baed7b3e2267c9e1: Deleting consensus metadata
I20260928 04:53:34.599923 2036 ts_tablet_manager.cc:1918] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:53:34.600023 2036 log.cc:1196] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/wal/wals/aea41e78d38f496586819440ad0a2b35
I20260928 04:53:34.600597 2036 ts_tablet_manager.cc:1939] T aea41e78d38f496586819440ad0a2b35 P f8630f596a3f4706801f0995c2e80917: Deleting consensus metadata
I20260928 04:53:34.600998 2013 tablet_bootstrap.cc:492] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1: No bootstrap required, opened a new log
I20260928 04:53:34.601073 2013 ts_tablet_manager.cc:1397] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:53:34.601235 2013 raft_consensus.cc:348] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } }
I20260928 04:53:34.601322 2013 raft_consensus.cc:374] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:34.601349 2013 raft_consensus.cc:729] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d521c1f2ed498aa1f533997a1b48e1, State: Initialized, Role: FOLLOWER
I20260928 04:53:34.601406 2013 consensus_queue.cc:260] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } }
I20260928 04:53:34.601502 2013 ts_tablet_manager.cc:1428] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:34.605597 1492 catalog_manager.cc:4841] TS 2bef0189d9fe4ae7baed7b3e2267c9e1 (127.0.98.2:41413): tablet aea41e78d38f496586819440ad0a2b35 (table test-table [id=af8c0f61ff7442d3b64ea8669dc4d493]) successfully deleted
I20260928 04:53:34.608714 2033 raft_consensus.cc:482] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:53:34.608855 2033 raft_consensus.cc:504] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } }
I20260928 04:53:34.609148 2033 leader_election.cc:288] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8630f596a3f4706801f0995c2e80917 (127.0.98.1:41025), 63d521c1f2ed498aa1f533997a1b48e1 (127.0.98.3:32943)
I20260928 04:53:34.609468 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af9fa78a98c42f5a68d17b34f104bd9" candidate_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d521c1f2ed498aa1f533997a1b48e1" is_pre_election: true
I20260928 04:53:34.609496 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af9fa78a98c42f5a68d17b34f104bd9" candidate_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8630f596a3f4706801f0995c2e80917" is_pre_election: true
I20260928 04:53:34.609575 1944 raft_consensus.cc:2454] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bef0189d9fe4ae7baed7b3e2267c9e1 in term 0.
I20260928 04:53:34.609580 1664 raft_consensus.cc:2454] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bef0189d9fe4ae7baed7b3e2267c9e1 in term 0.
I20260928 04:53:34.609730 1735 leader_election.cc:302] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bef0189d9fe4ae7baed7b3e2267c9e1, f8630f596a3f4706801f0995c2e80917; no voters:
I20260928 04:53:34.609828 2033 raft_consensus.cc:2781] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:53:34.609870 2033 raft_consensus.cc:482] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:53:34.609894 2033 raft_consensus.cc:3037] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:34.611488 1491 catalog_manager.cc:4841] TS f8630f596a3f4706801f0995c2e80917 (127.0.98.1:41025): tablet aea41e78d38f496586819440ad0a2b35 (table test-table [id=af8c0f61ff7442d3b64ea8669dc4d493]) successfully deleted
I20260928 04:53:34.622773 1494 catalog_manager.cc:4841] TS 63d521c1f2ed498aa1f533997a1b48e1 (127.0.98.3:32943): tablet aea41e78d38f496586819440ad0a2b35 (table test-table [id=af8c0f61ff7442d3b64ea8669dc4d493]) successfully deleted
I20260928 04:53:34.684963 2033 raft_consensus.cc:504] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } }
I20260928 04:53:34.685225 2033 leader_election.cc:288] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [CANDIDATE]: Term 1 election: Requested vote from peers f8630f596a3f4706801f0995c2e80917 (127.0.98.1:41025), 63d521c1f2ed498aa1f533997a1b48e1 (127.0.98.3:32943)
I20260928 04:53:34.685477 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af9fa78a98c42f5a68d17b34f104bd9" candidate_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8630f596a3f4706801f0995c2e80917"
I20260928 04:53:34.685468 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af9fa78a98c42f5a68d17b34f104bd9" candidate_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d521c1f2ed498aa1f533997a1b48e1"
I20260928 04:53:34.685576 1664 raft_consensus.cc:3037] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:34.685592 1944 raft_consensus.cc:3037] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:34.686542 1664 raft_consensus.cc:2454] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bef0189d9fe4ae7baed7b3e2267c9e1 in term 1.
I20260928 04:53:34.686626 1944 raft_consensus.cc:2454] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bef0189d9fe4ae7baed7b3e2267c9e1 in term 1.
I20260928 04:53:34.686782 1735 leader_election.cc:302] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bef0189d9fe4ae7baed7b3e2267c9e1, f8630f596a3f4706801f0995c2e80917; no voters:
I20260928 04:53:34.686919 2033 raft_consensus.cc:2781] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:34.686969 2033 raft_consensus.cc:686] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 1 LEADER]: Becoming Leader. State: Replica: 2bef0189d9fe4ae7baed7b3e2267c9e1, State: Running, Role: LEADER
I20260928 04:53:34.687036 2033 consensus_queue.cc:237] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } }
I20260928 04:53:34.687784 1507 catalog_manager.cc:5495] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bef0189d9fe4ae7baed7b3e2267c9e1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } health_report { overall_health: UNKNOWN } } }
Sep 28 04:53:34 dist-test-slave-m39l krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:34.750988 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.743602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54226 (local address 127.0.98.62:46551)
0928 04:53:34.743647 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:34.743649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:34.743798 (+ 149us) server_negotiation.cc:407] Connection header received
0928 04:53:34.743887 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:34.743888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:34.743898 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:34.743955 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:34.744473 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.744970 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.745752 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.745940 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.746023 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.748419 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:34.748420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:34.748421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:34.748429 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:34.749703 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:34.750103 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:34.750103 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:34.750104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:34.750171 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:34.750368 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:34.750368 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:34.750369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:34.750543 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:34.750608 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:34.750773 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 04:53:34.750861 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:53:34.751075 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.743539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46551 (local address 127.0.0.1:54226)
0928 04:53:34.743701 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:34.743714 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:34.743824 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:34.743961 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:34.743962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:34.744319 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:34.744378 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.744382 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.745060 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.745061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.745651 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.745654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.746032 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.746033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.746084 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.746786 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:34.746791 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:34.748323 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:34.749801 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:34.749810 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:34.749813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:34.750012 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:34.750238 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:34.750239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:34.750239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:34.750306 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:34.750597 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:34.750605 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:34.750654 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 04:53:34.750749 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 04:53:34.750937 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":78,"thread_start_us":95,"threads_started":1}
I20260928 04:53:34.753311 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54226:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 04:53:34.753554 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:53:34.755185 1644 tablet_service.cc:1467] Processing CreateTablet for tablet c67d2f4c361b4f50a486f13f60b30f3b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a95552c1a2014b6fa371617f78c5a7ed]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:53:34.755162 1924 tablet_service.cc:1467] Processing CreateTablet for tablet c67d2f4c361b4f50a486f13f60b30f3b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a95552c1a2014b6fa371617f78c5a7ed]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:53:34.755190 1784 tablet_service.cc:1467] Processing CreateTablet for tablet c67d2f4c361b4f50a486f13f60b30f3b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a95552c1a2014b6fa371617f78c5a7ed]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:53:34.755362 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c67d2f4c361b4f50a486f13f60b30f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:34.755362 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c67d2f4c361b4f50a486f13f60b30f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:34.755368 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c67d2f4c361b4f50a486f13f60b30f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:34.757016 2013 tablet_bootstrap.cc:492] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1: Bootstrap starting.
I20260928 04:53:34.757122 2014 tablet_bootstrap.cc:492] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917: Bootstrap starting.
I20260928 04:53:34.757481 2013 tablet_bootstrap.cc:654] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:34.757539 2014 tablet_bootstrap.cc:654] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:34.757860 2017 tablet_bootstrap.cc:492] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1: Bootstrap starting.
I20260928 04:53:34.758240 2017 tablet_bootstrap.cc:654] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:34.758394 2013 tablet_bootstrap.cc:492] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1: No bootstrap required, opened a new log
I20260928 04:53:34.758467 2013 ts_tablet_manager.cc:1397] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:34.758664 2013 raft_consensus.cc:348] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } }
I20260928 04:53:34.758762 2013 raft_consensus.cc:374] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:34.758793 2013 raft_consensus.cc:729] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d521c1f2ed498aa1f533997a1b48e1, State: Initialized, Role: FOLLOWER
I20260928 04:53:34.758864 2013 consensus_queue.cc:260] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } }
I20260928 04:53:34.758971 2013 ts_tablet_manager.cc:1428] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:34.759066 2017 tablet_bootstrap.cc:492] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1: No bootstrap required, opened a new log
I20260928 04:53:34.759115 2017 ts_tablet_manager.cc:1397] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:34.759150 2019 tablet_replica.cc:439] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } }
I20260928 04:53:34.759227 2017 raft_consensus.cc:348] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } }
I20260928 04:53:34.759245 2019 tablet_replica.cc:442] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:53:34.759289 2017 raft_consensus.cc:374] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:34.759320 2017 raft_consensus.cc:729] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bef0189d9fe4ae7baed7b3e2267c9e1, State: Initialized, Role: FOLLOWER
I20260928 04:53:34.759367 2017 consensus_queue.cc:260] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } }
I20260928 04:53:34.759464 2017 ts_tablet_manager.cc:1428] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:53:34.759580 2033 tablet_replica.cc:439] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } }
I20260928 04:53:34.759634 2014 tablet_bootstrap.cc:492] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917: No bootstrap required, opened a new log
I20260928 04:53:34.759649 2033 tablet_replica.cc:442] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:53:34.759681 2014 ts_tablet_manager.cc:1397] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 04:53:34.759826 2014 raft_consensus.cc:348] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } }
I20260928 04:53:34.759934 2014 raft_consensus.cc:374] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:34.759958 2014 raft_consensus.cc:729] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8630f596a3f4706801f0995c2e80917, State: Initialized, Role: FOLLOWER
I20260928 04:53:34.760030 2014 consensus_queue.cc:260] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } }
I20260928 04:53:34.760138 2014 ts_tablet_manager.cc:1428] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:34.760221 2018 tablet_replica.cc:439] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } }
I20260928 04:53:34.760360 2018 tablet_replica.cc:442] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 04:53:34.840324 1991 tablet.cc:2369] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:53:34.963842 2019 raft_consensus.cc:482] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:53:34.964035 2019 raft_consensus.cc:504] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } }
I20260928 04:53:34.964494 2019 leader_election.cc:288] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8630f596a3f4706801f0995c2e80917 (127.0.98.1:41025), 2bef0189d9fe4ae7baed7b3e2267c9e1 (127.0.98.2:41413)
I20260928 04:53:34.967888 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.964809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52187 (local address 127.0.98.2:41413)
0928 04:53:34.964865 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:34.964883 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:34.964956 (+ 73us) server_negotiation.cc:407] Connection header received
0928 04:53:34.965161 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:34.965162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:34.965172 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:34.965247 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:34.965442 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.965941 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.967134 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.967517 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.967604 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.967615 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:34.967780 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 04:53:34.967814 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:53:34.967912 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.964723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41413 (local address 127.0.98.3:52187)
0928 04:53:34.964853 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:34.964865 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:34.964972 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:34.965265 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:34.965266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:34.965269 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:34.965366 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.965368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.966043 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.966050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.967027 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.967030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.967606 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.967607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.967684 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.967687 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:53:34.967767 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 04:53:34.967814 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":47,"thread_start_us":63,"threads_started":1}
I20260928 04:53:34.968163 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.964730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41025 (local address 127.0.98.3:40489)
0928 04:53:34.964955 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:34.964966 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:34.965083 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:34.965257 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:34.965258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:34.965260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:34.965354 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.965356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.966105 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.966106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.967087 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.967089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.967834 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.967836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.967918 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.967922 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:53:34.968002 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 04:53:34.968054 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":70,"threads_started":1}
I20260928 04:53:34.968158 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67d2f4c361b4f50a486f13f60b30f3b" candidate_uuid: "63d521c1f2ed498aa1f533997a1b48e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" is_pre_election: true
I20260928 04:53:34.968330 1804 raft_consensus.cc:2454] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d521c1f2ed498aa1f533997a1b48e1 in term 0.
I20260928 04:53:34.968528 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.964885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40489 (local address 127.0.98.1:41025)
0928 04:53:34.964937 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:34.964939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:34.965068 (+ 129us) server_negotiation.cc:407] Connection header received
0928 04:53:34.965163 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:34.965164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:34.965179 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:34.965252 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:34.965485 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.966003 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.967208 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.967704 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.967819 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.967832 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:34.968384 (+ 552us) server_negotiation.cc:299] Negotiation successful
0928 04:53:34.968468 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 04:53:34.968531 1876 leader_election.cc:302] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bef0189d9fe4ae7baed7b3e2267c9e1, 63d521c1f2ed498aa1f533997a1b48e1; no voters:
I20260928 04:53:34.968693 2019 raft_consensus.cc:2781] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:53:34.968746 2019 raft_consensus.cc:482] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:53:34.968771 2019 raft_consensus.cc:3037] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:34.968799 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67d2f4c361b4f50a486f13f60b30f3b" candidate_uuid: "63d521c1f2ed498aa1f533997a1b48e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8630f596a3f4706801f0995c2e80917" is_pre_election: true
I20260928 04:53:34.968891 1664 raft_consensus.cc:2454] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d521c1f2ed498aa1f533997a1b48e1 in term 0.
I20260928 04:53:34.969594 2019 raft_consensus.cc:504] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } }
I20260928 04:53:34.969761 2019 leader_election.cc:288] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [CANDIDATE]: Term 1 election: Requested vote from peers f8630f596a3f4706801f0995c2e80917 (127.0.98.1:41025), 2bef0189d9fe4ae7baed7b3e2267c9e1 (127.0.98.2:41413)
I20260928 04:53:34.969910 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67d2f4c361b4f50a486f13f60b30f3b" candidate_uuid: "63d521c1f2ed498aa1f533997a1b48e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8630f596a3f4706801f0995c2e80917"
I20260928 04:53:34.969959 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67d2f4c361b4f50a486f13f60b30f3b" candidate_uuid: "63d521c1f2ed498aa1f533997a1b48e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1"
I20260928 04:53:34.970052 1804 raft_consensus.cc:3037] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:34.970052 1664 raft_consensus.cc:3037] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:34.970753 1664 raft_consensus.cc:2454] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d521c1f2ed498aa1f533997a1b48e1 in term 1.
I20260928 04:53:34.970945 1875 leader_election.cc:302] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d521c1f2ed498aa1f533997a1b48e1, f8630f596a3f4706801f0995c2e80917; no voters:
I20260928 04:53:34.970964 1804 raft_consensus.cc:2454] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d521c1f2ed498aa1f533997a1b48e1 in term 1.
I20260928 04:53:34.971069 2019 raft_consensus.cc:2781] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:34.971228 2019 raft_consensus.cc:686] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [term 1 LEADER]: Becoming Leader. State: Replica: 63d521c1f2ed498aa1f533997a1b48e1, State: Running, Role: LEADER
I20260928 04:53:34.971304 2019 consensus_queue.cc:237] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } }
I20260928 04:53:34.971740 2054 tablet_replica.cc:439] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1: TxnStatusTablet state changed. Reason: New leader 63d521c1f2ed498aa1f533997a1b48e1. Latest consensus state: current_term: 1 leader_uuid: "63d521c1f2ed498aa1f533997a1b48e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } }
I20260928 04:53:34.971866 2054 tablet_replica.cc:442] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:53:34.972132 2056 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 04:53:34.972165 1506 catalog_manager.cc:5495] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63d521c1f2ed498aa1f533997a1b48e1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "63d521c1f2ed498aa1f533997a1b48e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } health_report { overall_health: HEALTHY } } }
I20260928 04:53:34.972260 2056 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 04:53:34.989924 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.987252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32943 (local address 127.0.0.1:47992)
0928 04:53:34.987356 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:34.987369 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:34.987424 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:34.987629 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:34.987630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:34.987631 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:53:34.987692 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.987694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.988406 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.988407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.988982 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.988984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.989364 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.989365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.989430 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.989445 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:53:34.989737 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:53:34.989746 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 04:53:34.989798 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 04:53:34.989848 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 04:53:34.989933 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.987326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47992 (local address 127.0.98.3:32943)
0928 04:53:34.987444 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:34.987446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:34.987456 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:53:34.987494 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:34.987494 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:34.987526 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:34.987625 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:53:34.987776 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.988316 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.989073 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.989279 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.989352 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:34.989558 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:53:34.989661 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:53:34.989741 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:34.989803 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 04:53:34.989824 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":47,"threads_started":1}
I20260928 04:53:34.991081 1664 raft_consensus.cc:1261] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917 [term 1 FOLLOWER]: Refusing update from remote peer 63d521c1f2ed498aa1f533997a1b48e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:34.991207 1804 raft_consensus.cc:1261] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 1 FOLLOWER]: Refusing update from remote peer 63d521c1f2ed498aa1f533997a1b48e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:34.991346 2054 consensus_queue.cc:1035] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:34.991460 2019 consensus_queue.cc:1035] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:34.992162 2033 tablet_replica.cc:439] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1: TxnStatusTablet state changed. Reason: New leader 63d521c1f2ed498aa1f533997a1b48e1. Latest consensus state: current_term: 1 leader_uuid: "63d521c1f2ed498aa1f533997a1b48e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } }
I20260928 04:53:34.992266 2033 tablet_replica.cc:442] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:53:34.992273 2018 tablet_replica.cc:439] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917: TxnStatusTablet state changed. Reason: New leader 63d521c1f2ed498aa1f533997a1b48e1. Latest consensus state: current_term: 1 leader_uuid: "63d521c1f2ed498aa1f533997a1b48e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } }
I20260928 04:53:34.992380 2018 tablet_replica.cc:442] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:53:34.993176 2054 tablet_replica.cc:439] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63d521c1f2ed498aa1f533997a1b48e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } }
I20260928 04:53:34.993260 2054 tablet_replica.cc:442] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:53:34.993968 2018 tablet_replica.cc:439] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63d521c1f2ed498aa1f533997a1b48e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } }
I20260928 04:53:34.994050 2018 tablet_replica.cc:442] T c67d2f4c361b4f50a486f13f60b30f3b P f8630f596a3f4706801f0995c2e80917: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:53:34.994405 2054 tablet_replica.cc:439] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63d521c1f2ed498aa1f533997a1b48e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } }
I20260928 04:53:34.994529 2054 tablet_replica.cc:442] T c67d2f4c361b4f50a486f13f60b30f3b P 63d521c1f2ed498aa1f533997a1b48e1: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:53:34.995007 2033 tablet_replica.cc:439] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63d521c1f2ed498aa1f533997a1b48e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 } } peers { permanent_uuid: "2bef0189d9fe4ae7baed7b3e2267c9e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41413 } } peers { permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 } } }
I20260928 04:53:34.995100 2033 tablet_replica.cc:442] T c67d2f4c361b4f50a486f13f60b30f3b P 2bef0189d9fe4ae7baed7b3e2267c9e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:53:35.000723 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.998008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52609 (local address 127.0.98.3:32943)
0928 04:53:34.998076 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:34.998078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:34.998122 (+ 44us) server_negotiation.cc:407] Connection header received
0928 04:53:34.998251 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:34.998252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:34.998265 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:34.998357 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:53:34.998550 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.999059 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.999921 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:35.000139 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:35.000232 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:35.000460 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:53:35.000491 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:53:35.000555 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:35.000614 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 04:53:35.000639 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 04:53:35.000752 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:34.997849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32943 (local address 127.0.98.2:52609)
0928 04:53:34.998050 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:34.998065 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:34.998145 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:34.998365 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:34.998366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:34.998369 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:53:34.998429 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.998431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:34.999157 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:34.999159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:34.999816 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:34.999819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:35.000236 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:35.000237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:35.000310 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:35.000338 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:53:35.000555 (+ 217us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:53:35.000558 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:53:35.000613 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 04:53:35.000668 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
I20260928 04:53:35.003830 1664 raft_consensus.cc:1261] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917 [term 1 FOLLOWER]: Refusing update from remote peer 2bef0189d9fe4ae7baed7b3e2267c9e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:35.003880 1944 raft_consensus.cc:1261] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1 [term 1 FOLLOWER]: Refusing update from remote peer 2bef0189d9fe4ae7baed7b3e2267c9e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:35.004094 2033 consensus_queue.cc:1035] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8630f596a3f4706801f0995c2e80917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:35.004232 2023 consensus_queue.cc:1035] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63d521c1f2ed498aa1f533997a1b48e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:35.018913 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:35.014980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41413 (local address 127.0.98.3:52425)
0928 04:53:35.015137 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:35.015150 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:35.015221 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:35.015367 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:35.015368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:35.015370 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:53:35.015452 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:35.015454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:35.016291 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:35.016292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:35.017893 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:35.017896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:35.018386 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:35.018387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:35.018453 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:35.018474 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:53:35.018715 (+ 241us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:53:35.018718 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:53:35.018762 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 04:53:35.018815 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260928 04:53:35.018888 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:35.015036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52425 (local address 127.0.98.2:41413)
0928 04:53:35.015107 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:35.015109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:35.015202 (+ 93us) server_negotiation.cc:407] Connection header received
0928 04:53:35.015286 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:35.015287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:35.015299 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:35.015352 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:53:35.015519 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:35.016188 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:35.018042 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:35.018283 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:35.018350 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:35.018616 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:53:35.018655 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:53:35.018710 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:35.018775 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 04:53:35.018799 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 04:53:35.027618 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:53:35.027751 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:53:35.028461 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5e9b63ad7374f9f89d756d23c504af5: Sending DeleteTablet for 3 replicas of tablet 8af9fa78a98c42f5a68d17b34f104bd9
I20260928 04:53:35.028834 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 8af9fa78a98c42f5a68d17b34f104bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51486
I20260928 04:53:35.028877 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 8af9fa78a98c42f5a68d17b34f104bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52312
I20260928 04:53:35.028975 2037 tablet_replica.cc:330] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1: stopping tablet replica
I20260928 04:53:35.028963 2036 tablet_replica.cc:330] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917: stopping tablet replica
I20260928 04:53:35.029026 2036 raft_consensus.cc:2229] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:53:35.029034 2037 raft_consensus.cc:2229] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:53:35.029071 2036 raft_consensus.cc:2258] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:35.029102 2037 raft_consensus.cc:2258] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:35.029441 2037 ts_tablet_manager.cc:1905] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:35.029451 2036 ts_tablet_manager.cc:1905] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:35.031052 2036 ts_tablet_manager.cc:1918] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:53:35.031136 2036 log.cc:1196] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-0/wal/wals/8af9fa78a98c42f5a68d17b34f104bd9
I20260928 04:53:35.031360 2036 ts_tablet_manager.cc:1939] T 8af9fa78a98c42f5a68d17b34f104bd9 P f8630f596a3f4706801f0995c2e80917: Deleting consensus metadata
I20260928 04:53:35.031383 2037 ts_tablet_manager.cc:1918] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:53:35.031479 2037 log.cc:1196] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/8af9fa78a98c42f5a68d17b34f104bd9
I20260928 04:53:35.031749 2037 ts_tablet_manager.cc:1939] T 8af9fa78a98c42f5a68d17b34f104bd9 P 2bef0189d9fe4ae7baed7b3e2267c9e1: Deleting consensus metadata
I20260928 04:53:35.032150 1491 catalog_manager.cc:4841] TS f8630f596a3f4706801f0995c2e80917 (127.0.98.1:41025): tablet 8af9fa78a98c42f5a68d17b34f104bd9 (table test-table [id=f1be29d73fbb4061901ef76d5afd8860]) successfully deleted
I20260928 04:53:35.032169 1492 catalog_manager.cc:4841] TS 2bef0189d9fe4ae7baed7b3e2267c9e1 (127.0.98.2:41413): tablet 8af9fa78a98c42f5a68d17b34f104bd9 (table test-table [id=f1be29d73fbb4061901ef76d5afd8860]) successfully deleted
I20260928 04:53:35.032246 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 8af9fa78a98c42f5a68d17b34f104bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47984
I20260928 04:53:35.032362 2038 tablet_replica.cc:330] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1: stopping tablet replica
I20260928 04:53:35.032416 2038 raft_consensus.cc:2229] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:53:35.032462 2038 raft_consensus.cc:2258] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:35.032763 2038 ts_tablet_manager.cc:1905] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:35.034375 2038 ts_tablet_manager.cc:1918] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:53:35.034456 2038 log.cc:1196] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/8af9fa78a98c42f5a68d17b34f104bd9
I20260928 04:53:35.034698 2038 ts_tablet_manager.cc:1939] T 8af9fa78a98c42f5a68d17b34f104bd9 P 63d521c1f2ed498aa1f533997a1b48e1: Deleting consensus metadata
I20260928 04:53:35.035180 1494 catalog_manager.cc:4841] TS 63d521c1f2ed498aa1f533997a1b48e1 (127.0.98.3:32943): tablet 8af9fa78a98c42f5a68d17b34f104bd9 (table test-table [id=f1be29d73fbb4061901ef76d5afd8860]) successfully deleted
Sep 28 04:53:35 dist-test-slave-m39l krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571214, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:53:35.042624 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:35.032849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51488 (local address 127.0.98.1:41025)
0928 04:53:35.032905 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:35.032907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:35.033178 (+ 271us) server_negotiation.cc:407] Connection header received
0928 04:53:35.033229 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:35.033230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:35.033245 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:35.033318 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:35.034015 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:35.034934 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:35.035780 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:35.036132 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:35.036214 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:35.038995 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:35.039009 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:35.039010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:35.039019 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:35.041102 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:35.041519 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:35.041523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:35.041524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:35.041601 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:35.041851 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:35.041852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:35.041852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:35.042099 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:35.042316 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:35.042371 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 04:53:35.042456 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 04:53:35.043147 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51488
I20260928 04:53:35.050375 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:35.044311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.0.98.62:46551)
0928 04:53:35.044360 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:35.044362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:35.044372 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:53:35.044433 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:35.044434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:35.044443 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:35.044508 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:35.045005 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:35.045508 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:35.046199 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:35.046407 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:35.046519 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:35.047777 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:35.047779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:35.047779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:35.047787 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:35.049248 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:35.049609 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:35.049610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:35.049610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:35.049671 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:35.049866 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:35.049867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:35.049867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:35.049984 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:35.050095 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:35.050188 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 04:53:35.050268 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 04:53:35.050714 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54228
I20260928 04:53:35.051254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 1579
I20260928 04:53:35.058794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 1719
I20260928 04:53:35.065209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 1859
I20260928 04:53:35.070870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 1475
2026-09-28T04:53:35Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2599 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:53:35 dist-test-slave-m39l krb5kdc[2079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:53:35 dist-test-slave-m39l krb5kdc[2079](info): set up 2 sockets
Sep 28 04:53:35 dist-test-slave-m39l krb5kdc[2079](info): commencing operation
krb5kdc: starting...
W20260928 04:53:36.122275 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:53:36 dist-test-slave-m39l krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:53:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:53:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:53:36.252133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33749
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:36.338021 2092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:36.338256 2092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:36.338284 2092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:36.339591 2092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:53:36.339644 2092 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 04:53:36.339658 2092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:36.339672 2092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:36.339684 2092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:53:36.339697 2092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:53:36.341297 2092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33749
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:36.341517 2092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:36.341706 2092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:36.347275 2098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:36.347396 2092 server_base.cc:1034] running on GCE node
W20260928 04:53:36.347287 2100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:36.348037 2097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:36.348368 2092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:36.348716 2092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:36.349944 2092 hybrid_clock.cc:648] HybridClock initialized: now 1790571216349915 us; error 44 us; skew 500 ppm
Sep 28 04:53:36 dist-test-slave-m39l krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:36.353025 2092 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:53:36.353634 2092 webserver.cc:460] Webserver started at http://127.0.98.62:35129/ using document root <none> and password file <none>
I20260928 04:53:36.353920 2092 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:36.354003 2092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:36.354179 2092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:36.355470 2092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "55b5ad84fe754720a4a51dcbb2079b31"
format_stamp: "Formatted at 2026-09-28 04:53:36 on dist-test-slave-m39l"
server_key: "1758edb7a4cf900032c87a5072924930"
server_key_iv: "a6f8a1559a9024781dbfa36b1382452f"
server_key_version: "encryptionkey@0"
I20260928 04:53:36.355876 2092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "55b5ad84fe754720a4a51dcbb2079b31"
format_stamp: "Formatted at 2026-09-28 04:53:36 on dist-test-slave-m39l"
server_key: "1758edb7a4cf900032c87a5072924930"
server_key_iv: "a6f8a1559a9024781dbfa36b1382452f"
server_key_version: "encryptionkey@0"
I20260928 04:53:36.357492 2092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:53:36.358405 2107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:36.358690 2092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:36.358816 2092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "55b5ad84fe754720a4a51dcbb2079b31"
format_stamp: "Formatted at 2026-09-28 04:53:36 on dist-test-slave-m39l"
server_key: "1758edb7a4cf900032c87a5072924930"
server_key_iv: "a6f8a1559a9024781dbfa36b1382452f"
server_key_version: "encryptionkey@0"
I20260928 04:53:36.358981 2092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:36.369657 2092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:36.372493 2092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:36.372725 2092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:36.380203 2092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33749
I20260928 04:53:36.380282 2169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33749 every 8 connection(s)
I20260928 04:53:36.380708 2092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:53:36.381738 2170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:36.384356 2170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31: Bootstrap starting.
I20260928 04:53:36.385011 2170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:36.385320 2170 log.cc:824] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:36.386046 2170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31: No bootstrap required, opened a new log
I20260928 04:53:36.388134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 2092
I20260928 04:53:36.388274 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:53:36.388458 392 external_mini_cluster.cc:1428] Setting key 3d72c79d8ee5ba2a18e2507a58b8631a
I20260928 04:53:36.388314 2170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b5ad84fe754720a4a51dcbb2079b31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33749 } }
I20260928 04:53:36.388509 2170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:36.388574 2170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55b5ad84fe754720a4a51dcbb2079b31, State: Initialized, Role: FOLLOWER
I20260928 04:53:36.388729 2170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b5ad84fe754720a4a51dcbb2079b31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33749 } }
I20260928 04:53:36.388815 2170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:53:36.388862 2170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:53:36.388942 2170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:36.389607 2170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b5ad84fe754720a4a51dcbb2079b31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33749 } }
I20260928 04:53:36.389735 2170 leader_election.cc:302] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55b5ad84fe754720a4a51dcbb2079b31; no voters:
I20260928 04:53:36.389927 2170 leader_election.cc:288] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:53:36.389962 2175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:36.390064 2175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31 [term 1 LEADER]: Becoming Leader. State: Replica: 55b5ad84fe754720a4a51dcbb2079b31, State: Running, Role: LEADER
I20260928 04:53:36.390149 2175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b5ad84fe754720a4a51dcbb2079b31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33749 } }
I20260928 04:53:36.390214 2170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:36.390698 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55b5ad84fe754720a4a51dcbb2079b31. Latest consensus state: current_term: 1 leader_uuid: "55b5ad84fe754720a4a51dcbb2079b31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b5ad84fe754720a4a51dcbb2079b31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33749 } } }
I20260928 04:53:36.390822 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:36.390686 2176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55b5ad84fe754720a4a51dcbb2079b31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b5ad84fe754720a4a51dcbb2079b31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33749 } } }
I20260928 04:53:36.391121 2176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:36.391153 2184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:53:36.391716 2184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:53:36.394039 2184 catalog_manager.cc:1347] Generated new cluster ID: b38ccd3e255b4e9eabb20ba6bb36c302
I20260928 04:53:36.394109 2184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:53:36 dist-test-slave-m39l krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:36.402196 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:36.389042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.0.98.62:33749)
0928 04:53:36.389259 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:36.389274 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:36.389301 (+ 27us) server_negotiation.cc:407] Connection header received
0928 04:53:36.389406 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:36.389412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:36.389457 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:36.389596 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:36.390427 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.392436 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.393554 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.394096 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.394214 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:36.397376 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:36.397382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:36.397383 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:36.397399 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:36.400318 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:36.400813 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:36.400817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:36.400817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:36.400969 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:36.401203 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:36.401203 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:36.401204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:36.401542 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:36.401694 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:36.401831 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 04:53:36.401970 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":97,"threads_started":1}
I20260928 04:53:36.416863 2184 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:53:36.417529 2184 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:53:36.426180 2184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31: Generated new TSK 0
I20260928 04:53:36.426445 2184 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:53:36.462709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33749
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:36.563429 2196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:36.563679 2196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:36.563710 2196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:36.565289 2196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:36.565371 2196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:36.565398 2196 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 04:53:36.565410 2196 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:53:36.565429 2196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:53:36.567734 2196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:36.568192 2196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:36.568509 2196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:36.574896 2201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:36.575266 2204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:36.575562 2202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:36.576479 2196 server_base.cc:1034] running on GCE node
I20260928 04:53:36.576877 2196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:36.577178 2196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:36.578483 2196 hybrid_clock.cc:648] HybridClock initialized: now 1790571216578440 us; error 47 us; skew 500 ppm
Sep 28 04:53:36 dist-test-slave-m39l krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:36.581528 2196 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:53:36.582000 2196 webserver.cc:460] Webserver started at http://127.0.98.1:37813/ using document root <none> and password file <none>
I20260928 04:53:36.582214 2196 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:36.582252 2196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:36.582351 2196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:36.584174 2196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e"
format_stamp: "Formatted at 2026-09-28 04:53:36 on dist-test-slave-m39l"
server_key: "9c269ea38d5bc23ee522297145b358ab"
server_key_iv: "3a5da575060dee7401ecc3ba65a3196b"
server_key_version: "encryptionkey@0"
I20260928 04:53:36.584573 2196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e"
format_stamp: "Formatted at 2026-09-28 04:53:36 on dist-test-slave-m39l"
server_key: "9c269ea38d5bc23ee522297145b358ab"
server_key_iv: "3a5da575060dee7401ecc3ba65a3196b"
server_key_version: "encryptionkey@0"
I20260928 04:53:36.586141 2196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:53:36.587831 2211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:36.588274 2196 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:36.588524 2196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e"
format_stamp: "Formatted at 2026-09-28 04:53:36 on dist-test-slave-m39l"
server_key: "9c269ea38d5bc23ee522297145b358ab"
server_key_iv: "3a5da575060dee7401ecc3ba65a3196b"
server_key_version: "encryptionkey@0"
I20260928 04:53:36.588840 2196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:36.600800 2196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:36.604542 2196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:36.604743 2196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:36.605423 2196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:36.605499 2196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:36.605544 2196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:36.605573 2196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:36.614357 2196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43407
I20260928 04:53:36.614537 2326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43407 every 8 connection(s)
I20260928 04:53:36.614852 2196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:53:36 dist-test-slave-m39l krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:36.619479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 2196
I20260928 04:53:36.619611 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:53:36.619803 392 external_mini_cluster.cc:1428] Setting key b60cb489a771e814cf08035b6f997281
I20260928 04:53:36.620378 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:36.606100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53055 (local address 127.0.98.62:33749)
0928 04:53:36.606162 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:36.606164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:36.606758 (+ 594us) server_negotiation.cc:407] Connection header received
0928 04:53:36.607328 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:36.607330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:36.607342 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:36.607402 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:36.608764 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.609615 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.611499 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.611783 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.611856 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:36.615602 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:36.615604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:36.615605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:36.615621 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:36.618049 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:36.618642 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:36.618643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:36.618643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:36.618704 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:36.619327 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:36.619329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:36.619330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:36.619544 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:36.619636 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:36.620155 (+ 519us) server_negotiation.cc:299] Negotiation successful
0928 04:53:36.620274 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 04:53:36.620553 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:36.606134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33749 (local address 127.0.98.1:53055)
0928 04:53:36.606597 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:36.606653 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:36.607210 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:36.607487 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:36.607501 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:36.608256 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:36.608658 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:36.608667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.609733 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.609735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:36.611338 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:36.611342 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.611899 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.611900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:36.612085 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:36.613031 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:36.613054 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:36.615497 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:36.618186 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:36.618196 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:36.618198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:36.618483 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:36.618818 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:36.618819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:36.618819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:36.619041 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:36.619666 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:36.619674 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:36.619947 (+ 273us) client_negotiation.cc:769] Sending connection context
0928 04:53:36.620048 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 04:53:36.620286 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":302,"threads_started":1}
I20260928 04:53:36.621891 2327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33749
I20260928 04:53:36.622017 2327 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:36.622236 2327 heartbeater.cc:511] Master 127.0.98.62:33749 requested a full tablet report, sending...
I20260928 04:53:36.623255 2124 ts_manager.cc:194] Registered new tserver with Master: 4b0efa87f59a4e3bb494c0a505d6bc6e (127.0.98.1:43407)
I20260928 04:53:36.624523 2124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53055
I20260928 04:53:36.631767 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:36.623202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52979 (local address 127.0.98.62:33749)
0928 04:53:36.623300 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:36.623302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:36.623819 (+ 517us) server_negotiation.cc:407] Connection header received
0928 04:53:36.623841 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:36.623842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:36.623851 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:36.623919 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:36.624279 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.624987 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.626581 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.626955 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.627039 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:36.628532 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:36.628533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:36.628534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:36.628541 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:36.630012 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:36.630482 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:36.630483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:36.630483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:36.630607 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:36.630903 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:36.630904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:36.630905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:36.631152 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:36.631216 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:36.631612 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 04:53:36.631697 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 04:53:36.631812 2334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:36.623157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33749 (local address 127.0.98.1:52979)
0928 04:53:36.623420 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:36.623431 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:36.623612 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:36.623938 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:36.623939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:36.624144 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:36.624199 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:36.624201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.625087 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.625089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:36.626314 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:36.626323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.627069 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.627071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:36.627208 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:36.627800 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:36.627808 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:36.628379 (+ 571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:36.630118 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:36.630120 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:36.630120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:36.630380 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:36.630695 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:36.630696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:36.630697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:36.630783 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:36.631242 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:36.631243 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:36.631309 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 04:53:36.631542 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 04:53:36.631685 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":19,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:53:36.669582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33749
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:36.769384 2336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:36.769690 2336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:36.769738 2336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:36.771284 2336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:36.771394 2336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:36.771445 2336 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 04:53:36.771488 2336 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:53:36.771522 2336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:53:36.773124 2336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:36.773435 2336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:36.773682 2336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:36.778939 2341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:36.778966 2344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:36.778951 2342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:36.779176 2336 server_base.cc:1034] running on GCE node
I20260928 04:53:36.779452 2336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:36.779697 2336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:36.780915 2336 hybrid_clock.cc:648] HybridClock initialized: now 1790571216780892 us; error 58 us; skew 500 ppm
Sep 28 04:53:36 dist-test-slave-m39l krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:36.787568 2336 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:53:36.788070 2336 webserver.cc:460] Webserver started at http://127.0.98.2:41285/ using document root <none> and password file <none>
I20260928 04:53:36.788290 2336 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:36.788336 2336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:36.788431 2336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:36.789505 2336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "a288da0acaa54759aaf79bbaab0e0a52"
format_stamp: "Formatted at 2026-09-28 04:53:36 on dist-test-slave-m39l"
server_key: "aebf9f47140897ad4d6c84404a492a3a"
server_key_iv: "f40531a0c9a5046b6fef9a598c188d62"
server_key_version: "encryptionkey@0"
I20260928 04:53:36.789878 2336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a288da0acaa54759aaf79bbaab0e0a52"
format_stamp: "Formatted at 2026-09-28 04:53:36 on dist-test-slave-m39l"
server_key: "aebf9f47140897ad4d6c84404a492a3a"
server_key_iv: "f40531a0c9a5046b6fef9a598c188d62"
server_key_version: "encryptionkey@0"
I20260928 04:53:36.791620 2336 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 04:53:36.792474 2351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:36.792680 2336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:53:36.792794 2336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "a288da0acaa54759aaf79bbaab0e0a52"
format_stamp: "Formatted at 2026-09-28 04:53:36 on dist-test-slave-m39l"
server_key: "aebf9f47140897ad4d6c84404a492a3a"
server_key_iv: "f40531a0c9a5046b6fef9a598c188d62"
server_key_version: "encryptionkey@0"
I20260928 04:53:36.792883 2336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:36.807842 2336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:36.810004 2336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:36.810179 2336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:36.810793 2336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:36.810866 2336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:36.810926 2336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:36.810956 2336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 04:53:36 dist-test-slave-m39l krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:36.818063 2336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39893
I20260928 04:53:36.818146 2466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39893 every 8 connection(s)
I20260928 04:53:36.818549 2336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:53:36.821568 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:36.811355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47993 (local address 127.0.98.62:33749)
0928 04:53:36.811428 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:36.811430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:36.811664 (+ 234us) server_negotiation.cc:407] Connection header received
0928 04:53:36.812283 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:36.812284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:36.812298 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:36.812384 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:36.813337 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.814190 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.815404 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.815622 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.815705 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:36.818560 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:36.818563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:36.818563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:36.818571 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:36.820209 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:36.820664 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:36.820664 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:36.820665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:36.820718 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:36.820946 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:36.820946 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:36.820947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:36.821055 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:36.821123 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:36.821427 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 04:53:36.821500 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:53:36.821755 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:36.811337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33749 (local address 127.0.98.2:47993)
0928 04:53:36.811520 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:36.811563 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:36.812137 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:36.812490 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:36.812511 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:36.812914 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:36.813217 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:36.813224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.814310 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.814311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:36.815306 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:36.815308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.815731 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.815732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:36.815893 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:36.816487 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:36.816496 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:36.818372 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:36.820298 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:36.820304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:36.820306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:36.820588 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:36.820776 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:36.820776 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:36.820777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:36.820892 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:36.821095 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:36.821097 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:36.821278 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 04:53:36.821410 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 04:53:36.821542 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":117,"threads_started":1}
I20260928 04:53:36.822273 2467 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33749
I20260928 04:53:36.822362 2467 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:36.822644 2467 heartbeater.cc:511] Master 127.0.98.62:33749 requested a full tablet report, sending...
I20260928 04:53:36.823045 2123 ts_manager.cc:194] Registered new tserver with Master: a288da0acaa54759aaf79bbaab0e0a52 (127.0.98.2:39893)
I20260928 04:53:36.823482 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47993
I20260928 04:53:36.826157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 2336
I20260928 04:53:36.826292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:53:36.826594 392 external_mini_cluster.cc:1428] Setting key 8495b56d3e22bd876746ae6a60630010
I20260928 04:53:36.829672 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:36.823453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47683 (local address 127.0.98.62:33749)
0928 04:53:36.823498 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:36.823500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:36.823645 (+ 145us) server_negotiation.cc:407] Connection header received
0928 04:53:36.823722 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:36.823723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:36.823733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:36.823790 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:36.824131 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.824619 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.825317 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.825533 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.825610 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:36.826718 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:36.826720 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:36.826721 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:36.826729 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:36.828331 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:36.828693 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:36.828694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:36.828694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:36.828751 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:36.828997 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:36.828999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:36.829000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:36.829250 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:36.829332 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:36.829525 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 04:53:36.829598 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:53:36.829741 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:36.823403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33749 (local address 127.0.98.2:47683)
0928 04:53:36.823565 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:36.823576 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:36.823671 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:36.823795 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:36.823796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:36.823992 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:36.824053 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:36.824055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.824716 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.824717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:36.825234 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:36.825235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.825608 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.825609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:36.825684 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:36.826207 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:36.826212 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:36.826629 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:36.828429 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:36.828430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:36.828430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:36.828607 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:36.828827 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:36.828828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:36.828828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:36.828895 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:36.829335 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:36.829336 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:36.829389 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 04:53:36.829470 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 04:53:36.829597 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:53:36.854476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33749
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:36.947573 2476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:36.947813 2476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:36.947842 2476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:36.949026 2476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:36.949074 2476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:36.949097 2476 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 04:53:36.949111 2476 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:53:36.949131 2476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:53:36.950788 2476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:36.951002 2476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:36.951208 2476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:53:36.956904 2476 server_base.cc:1034] running on GCE node
W20260928 04:53:36.956944 2481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:36.956938 2482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:36.957176 2484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:36.957412 2476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:36.957702 2476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:36.958923 2476 hybrid_clock.cc:648] HybridClock initialized: now 1790571216958901 us; error 32 us; skew 500 ppm
Sep 28 04:53:36 dist-test-slave-m39l krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:36.961804 2476 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:53:36.962424 2476 webserver.cc:460] Webserver started at http://127.0.98.3:37213/ using document root <none> and password file <none>
I20260928 04:53:36.962731 2476 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:36.962823 2476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:36.963011 2476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:36.964218 2476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "6c6d4980832e4ce699d9261b17a700a2"
format_stamp: "Formatted at 2026-09-28 04:53:36 on dist-test-slave-m39l"
server_key: "ea17a65385b4f80fe6327f719e8d7241"
server_key_iv: "106e321959a2585deed268db50e3f4fa"
server_key_version: "encryptionkey@0"
I20260928 04:53:36.964629 2476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6c6d4980832e4ce699d9261b17a700a2"
format_stamp: "Formatted at 2026-09-28 04:53:36 on dist-test-slave-m39l"
server_key: "ea17a65385b4f80fe6327f719e8d7241"
server_key_iv: "106e321959a2585deed268db50e3f4fa"
server_key_version: "encryptionkey@0"
I20260928 04:53:36.966146 2476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:53:36.967104 2491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:36.967331 2476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:53:36.967442 2476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "6c6d4980832e4ce699d9261b17a700a2"
format_stamp: "Formatted at 2026-09-28 04:53:36 on dist-test-slave-m39l"
server_key: "ea17a65385b4f80fe6327f719e8d7241"
server_key_iv: "106e321959a2585deed268db50e3f4fa"
server_key_version: "encryptionkey@0"
I20260928 04:53:36.967530 2476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:36.983175 2476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:36.985600 2476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:36.985777 2476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:36.986330 2476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:36.986397 2476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:36.986451 2476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:36.986509 2476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:36.994954 2476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44813
I20260928 04:53:36.995016 2606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44813 every 8 connection(s)
Sep 28 04:53:36 dist-test-slave-m39l krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:36.995424 2476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:53:36.999235 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:36.987039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47117 (local address 127.0.98.62:33749)
0928 04:53:36.987121 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:36.987124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:36.987381 (+ 257us) server_negotiation.cc:407] Connection header received
0928 04:53:36.988468 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:36.988469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:36.988481 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:36.988541 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:36.989735 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.990534 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.992322 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.992694 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.992774 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:36.995960 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:36.995961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:36.995962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:36.995970 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:36.997517 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:36.998022 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:36.998023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:36.998023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:36.998080 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:36.998450 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:36.998451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:36.998452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:36.998660 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:36.998729 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:36.999066 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 04:53:36.999167 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 04:53:36.999491 2500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:36.986987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33749 (local address 127.0.98.3:47117)
0928 04:53:36.987205 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:36.987274 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:36.988243 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:36.988662 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:36.988669 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:36.989239 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:36.989570 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:36.989576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.990650 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.990651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:36.992196 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:36.992201 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:36.992852 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:36.992853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:36.993032 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:36.993773 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:36.993783 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:36.995823 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:36.997615 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:36.997620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:36.997622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:36.997917 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:36.998174 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:36.998175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:36.998176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:36.998349 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:36.998751 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:36.998758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:36.998898 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 04:53:36.999002 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 04:53:36.999159 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":115,"threads_started":1}
I20260928 04:53:37.000154 2607 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33749
I20260928 04:53:37.000278 2607 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:37.000411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 2476
I20260928 04:53:37.000525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:53:37.000525 2607 heartbeater.cc:511] Master 127.0.98.62:33749 requested a full tablet report, sending...
I20260928 04:53:37.000753 392 external_mini_cluster.cc:1428] Setting key c03d8c79af9ed225cc18555bb4a7586b
I20260928 04:53:37.000905 2124 ts_manager.cc:194] Registered new tserver with Master: 6c6d4980832e4ce699d9261b17a700a2 (127.0.98.3:44813)
I20260928 04:53:37.001372 2124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47117
I20260928 04:53:37.002056 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:53:37.008899 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.001913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33749 (local address 127.0.98.3:44455)
0928 04:53:37.002085 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:37.002097 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:37.002179 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:37.002352 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:37.002352 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:37.002639 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:37.002699 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.002701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.003430 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.003431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.004299 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.004302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.004672 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.004673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.004758 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.005269 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:37.005274 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:37.005606 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:37.007566 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:37.007567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:37.007568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:37.007848 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:37.008120 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:37.008121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:37.008122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:37.008205 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:37.008508 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:37.008509 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:37.008573 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 04:53:37.008667 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 04:53:37.008806 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":98,"threads_started":1}
I20260928 04:53:37.008955 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.001926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44455 (local address 127.0.98.62:33749)
0928 04:53:37.001968 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:37.001971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:37.002151 (+ 180us) server_negotiation.cc:407] Connection header received
0928 04:53:37.002250 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:37.002251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:37.002262 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:37.002291 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:37.002795 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.003284 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.004398 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.004589 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.004650 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.005701 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:37.005702 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:37.005703 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:37.005712 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:37.007422 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:37.007959 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:37.007960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:37.007960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:37.008032 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:37.008277 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:37.008278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:37.008278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:37.008428 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:37.008488 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:37.008746 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 04:53:37.008861 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 04:53:37 dist-test-slave-m39l krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 04:53:37 dist-test-slave-m39l krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:37.029345 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.020943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.0.98.62:33749)
0928 04:53:37.021010 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:37.021016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:37.021025 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:53:37.021052 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:37.021053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:37.021062 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:37.021125 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:37.021941 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.022612 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.023615 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.023801 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.023865 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.026547 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:37.026550 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:37.026551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:37.026565 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:37.027968 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:37.028377 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:37.028378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:37.028385 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:37.028447 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:37.028730 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:37.028731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:37.028732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:37.028937 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:37.029029 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:37.029199 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 04:53:37.029272 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:53:37.031561 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:53:37.033412 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:53:37.040143 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.036063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43407 (local address 127.0.0.1:50802)
0928 04:53:37.036315 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:37.036360 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:37.036433 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:37.036645 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:37.036648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:37.036650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:37.036739 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.036742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.037818 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.037819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.038866 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.038868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.039841 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.039842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.039898 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.039908 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 04:53:37.039995 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 04:53:37.040048 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":66,"threads_started":1}
I20260928 04:53:37.040153 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.036161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.0.98.1:43407)
0928 04:53:37.036421 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:37.036426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:37.036437 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:53:37.036520 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:37.036522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:37.036537 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:37.036630 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:37.036835 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.037713 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.038965 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.039681 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.039743 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.039813 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:37.040010 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 04:53:37.040048 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":105,"threads_started":1}
I20260928 04:53:37.040894 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.036344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44813 (local address 127.0.0.1:41720)
0928 04:53:37.036700 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:37.036709 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:37.036743 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:37.037079 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:37.037080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:37.037082 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:37.037131 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.037133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.038519 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.038521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.039458 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.039460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.040610 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.040612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.040681 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.040686 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:53:37.040743 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 04:53:37.040782 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":26,"thread_start_us":53,"threads_started":1}
I20260928 04:53:37.040964 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.036140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39893 (local address 127.0.0.1:40636)
0928 04:53:37.036562 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:37.036574 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:37.036629 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:37.037129 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:37.037130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:37.037133 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:37.037174 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.037176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.038484 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.038485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.039642 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.039644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.040638 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.040639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.040706 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.040712 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 04:53:37.040773 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 04:53:37.040824 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":27,"thread_start_us":102,"threads_started":1}
I20260928 04:53:37.040979 2261 tablet_service.cc:1467] Processing CreateTablet for tablet 1c82544b516c465eba26c91197b6238d (DEFAULT_TABLE table=test-table [id=a3df18bbd7694d5abbf7fce8a2690cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:37.041311 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.036238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40636 (local address 127.0.98.2:39893)
0928 04:53:37.036953 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:37.036958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:37.036968 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:53:37.037006 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:37.037008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:37.037023 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:37.037126 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:37.037270 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.038389 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.039783 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.040544 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.041035 (+ 491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.041116 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:37.041147 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 04:53:37.041197 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":105,"threads_started":1}
I20260928 04:53:37.040925 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.036538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.0.98.3:44813)
0928 04:53:37.036726 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:37.036731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:37.036742 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:53:37.036813 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:37.036815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:37.036830 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:37.037007 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:37.037209 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.038389 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.039551 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.040516 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.040603 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.040660 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:37.040771 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 04:53:37.040813 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":84,"threads_started":1}
I20260928 04:53:37.041436 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c82544b516c465eba26c91197b6238d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:37.041901 2386 tablet_service.cc:1467] Processing CreateTablet for tablet 1c82544b516c465eba26c91197b6238d (DEFAULT_TABLE table=test-table [id=a3df18bbd7694d5abbf7fce8a2690cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:37.042028 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 1c82544b516c465eba26c91197b6238d (DEFAULT_TABLE table=test-table [id=a3df18bbd7694d5abbf7fce8a2690cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:37.042244 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c82544b516c465eba26c91197b6238d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:37.042392 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c82544b516c465eba26c91197b6238d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:37.045115 2630 tablet_bootstrap.cc:492] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52: Bootstrap starting.
I20260928 04:53:37.045575 2631 tablet_bootstrap.cc:492] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e: Bootstrap starting.
I20260928 04:53:37.046167 2630 tablet_bootstrap.cc:654] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:37.046222 2631 tablet_bootstrap.cc:654] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:37.046250 2632 tablet_bootstrap.cc:492] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2: Bootstrap starting.
I20260928 04:53:37.046586 2631 log.cc:824] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:37.046587 2630 log.cc:824] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:37.047261 2632 tablet_bootstrap.cc:654] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:37.047410 2631 tablet_bootstrap.cc:492] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e: No bootstrap required, opened a new log
I20260928 04:53:37.047410 2630 tablet_bootstrap.cc:492] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52: No bootstrap required, opened a new log
I20260928 04:53:37.047484 2631 ts_tablet_manager.cc:1397] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:53:37.047484 2630 ts_tablet_manager.cc:1397] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260928 04:53:37.047632 2632 log.cc:824] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:37.048386 2632 tablet_bootstrap.cc:492] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2: No bootstrap required, opened a new log
I20260928 04:53:37.048473 2632 ts_tablet_manager.cc:1397] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:53:37.049546 2631 raft_consensus.cc:348] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.049563 2630 raft_consensus.cc:348] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.049741 2631 raft_consensus.cc:374] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:37.049741 2630 raft_consensus.cc:374] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:37.049779 2631 raft_consensus.cc:729] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0efa87f59a4e3bb494c0a505d6bc6e, State: Initialized, Role: FOLLOWER
I20260928 04:53:37.049823 2630 raft_consensus.cc:729] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a288da0acaa54759aaf79bbaab0e0a52, State: Initialized, Role: FOLLOWER
I20260928 04:53:37.049952 2630 consensus_queue.cc:260] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.050005 2631 consensus_queue.cc:260] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.050272 2631 ts_tablet_manager.cc:1428] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 04:53:37.050314 2467 heartbeater.cc:503] Master 127.0.98.62:33749 was elected leader, sending a full tablet report...
I20260928 04:53:37.050416 2327 heartbeater.cc:503] Master 127.0.98.62:33749 was elected leader, sending a full tablet report...
I20260928 04:53:37.050869 2630 ts_tablet_manager.cc:1428] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 04:53:37.050829 2632 raft_consensus.cc:348] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.050992 2632 raft_consensus.cc:374] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:37.051036 2632 raft_consensus.cc:729] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c6d4980832e4ce699d9261b17a700a2, State: Initialized, Role: FOLLOWER
I20260928 04:53:37.051153 2632 consensus_queue.cc:260] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.051447 2632 ts_tablet_manager.cc:1428] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:53:37.051514 2607 heartbeater.cc:503] Master 127.0.98.62:33749 was elected leader, sending a full tablet report...
W20260928 04:53:37.069177 2468 tablet.cc:2369] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:53:37.115707 2328 tablet.cc:2369] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:53:37.119711 2638 raft_consensus.cc:482] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:53:37.119853 2638 raft_consensus.cc:504] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.120150 2638 leader_election.cc:288] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a288da0acaa54759aaf79bbaab0e0a52 (127.0.98.2:39893), 4b0efa87f59a4e3bb494c0a505d6bc6e (127.0.98.1:43407)
I20260928 04:53:37.123495 2500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.120376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39893 (local address 127.0.98.3:48583)
0928 04:53:37.120458 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:37.120473 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:37.120535 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:37.120735 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:37.120736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:37.120738 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:37.120810 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.120812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.121565 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.121567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.122522 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.122524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.123186 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.123187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.123266 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.123271 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:53:37.123348 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 04:53:37.123390 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 04:53:37.123529 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.120528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48583 (local address 127.0.98.2:39893)
0928 04:53:37.120575 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:37.120577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:37.120615 (+ 38us) server_negotiation.cc:407] Connection header received
0928 04:53:37.120645 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:37.120646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:37.120657 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:37.120728 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:37.120897 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.121461 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.122618 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.123103 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.123158 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.123168 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:37.123380 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 04:53:37.123425 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 04:53:37.123809 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.120529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58429 (local address 127.0.98.1:43407)
0928 04:53:37.120574 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:37.120576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:37.120726 (+ 150us) server_negotiation.cc:407] Connection header received
0928 04:53:37.120872 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:37.120873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:37.120883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:37.120943 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:37.121127 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.121708 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.122915 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.123418 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.123484 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.123493 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:37.123703 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 04:53:37.123747 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:53:37.123818 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.120375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43407 (local address 127.0.98.3:58429)
0928 04:53:37.120642 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:37.120654 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:37.120740 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:37.120941 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:37.120942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:37.120943 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:37.121012 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.121013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.121786 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.121787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.122792 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.122795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.123514 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.123515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.123593 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.123598 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:53:37.123690 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 04:53:37.123730 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":59,"thread_start_us":110,"threads_started":1}
I20260928 04:53:37.123936 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c82544b516c465eba26c91197b6238d" candidate_uuid: "6c6d4980832e4ce699d9261b17a700a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a288da0acaa54759aaf79bbaab0e0a52" is_pre_election: true
I20260928 04:53:37.124132 2421 raft_consensus.cc:2454] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c6d4980832e4ce699d9261b17a700a2 in term 0.
I20260928 04:53:37.124085 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c82544b516c465eba26c91197b6238d" candidate_uuid: "6c6d4980832e4ce699d9261b17a700a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" is_pre_election: true
I20260928 04:53:37.124228 2281 raft_consensus.cc:2454] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c6d4980832e4ce699d9261b17a700a2 in term 0.
I20260928 04:53:37.124368 2493 leader_election.cc:302] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c6d4980832e4ce699d9261b17a700a2, a288da0acaa54759aaf79bbaab0e0a52; no voters:
I20260928 04:53:37.124521 2638 raft_consensus.cc:2781] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:53:37.124612 2638 raft_consensus.cc:482] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:53:37.124648 2638 raft_consensus.cc:3037] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:37.125383 2638 raft_consensus.cc:504] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.125547 2638 leader_election.cc:288] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [CANDIDATE]: Term 1 election: Requested vote from peers a288da0acaa54759aaf79bbaab0e0a52 (127.0.98.2:39893), 4b0efa87f59a4e3bb494c0a505d6bc6e (127.0.98.1:43407)
I20260928 04:53:37.125663 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c82544b516c465eba26c91197b6238d" candidate_uuid: "6c6d4980832e4ce699d9261b17a700a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a288da0acaa54759aaf79bbaab0e0a52"
I20260928 04:53:37.125682 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c82544b516c465eba26c91197b6238d" candidate_uuid: "6c6d4980832e4ce699d9261b17a700a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e"
I20260928 04:53:37.125746 2421 raft_consensus.cc:3037] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:37.125782 2281 raft_consensus.cc:3037] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:37.126469 2281 raft_consensus.cc:2454] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c6d4980832e4ce699d9261b17a700a2 in term 1.
I20260928 04:53:37.126536 2421 raft_consensus.cc:2454] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c6d4980832e4ce699d9261b17a700a2 in term 1.
I20260928 04:53:37.127010 2494 leader_election.cc:302] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0efa87f59a4e3bb494c0a505d6bc6e, 6c6d4980832e4ce699d9261b17a700a2; no voters:
I20260928 04:53:37.127111 2638 raft_consensus.cc:2781] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:37.127249 2638 raft_consensus.cc:686] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [term 1 LEADER]: Becoming Leader. State: Replica: 6c6d4980832e4ce699d9261b17a700a2, State: Running, Role: LEADER
I20260928 04:53:37.127329 2638 consensus_queue.cc:237] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.128280 2123 catalog_manager.cc:5495] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c6d4980832e4ce699d9261b17a700a2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6c6d4980832e4ce699d9261b17a700a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } health_report { overall_health: HEALTHY } } }
Sep 28 04:53:37 dist-test-slave-m39l krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:37.158430 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.151587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50886 (local address 127.0.98.62:33749)
0928 04:53:37.151672 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:37.151675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:37.151758 (+ 83us) server_negotiation.cc:407] Connection header received
0928 04:53:37.151934 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:37.151935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:37.151945 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:37.152014 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:37.152478 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.153035 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.153732 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.153979 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.154044 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.156240 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:37.156242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:37.156242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:37.156249 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:37.157454 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:37.157806 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:37.157807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:37.157807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:37.157856 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:37.158034 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:37.158035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:37.158035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:37.158136 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:37.158186 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:37.158301 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 04:53:37.158364 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 04:53:37.158617 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.151505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33749 (local address 127.0.0.1:50886)
0928 04:53:37.151684 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:37.151696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:37.151790 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:37.152037 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:37.152038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:37.152341 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:37.152390 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.152392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.153119 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.153120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.153652 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.153654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.154054 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.154054 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.154097 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.154687 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:37.154691 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:37.156139 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:37.157536 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:37.157542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:37.157544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:37.157740 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:37.157913 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:37.157913 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:37.157914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:37.157967 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:37.158190 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:37.158195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:37.158236 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 04:53:37.158285 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 04:53:37.158417 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":43,"thread_start_us":111,"threads_started":1}
I20260928 04:53:37.160890 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50886:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 04:53:37.161146 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:53:37.162951 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 0907481421c240618e1b0ae2dcae7bf3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=96f5cec808c4467d9b1123c97ad32578]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:53:37.162951 2386 tablet_service.cc:1467] Processing CreateTablet for tablet 0907481421c240618e1b0ae2dcae7bf3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=96f5cec808c4467d9b1123c97ad32578]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:53:37.163126 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0907481421c240618e1b0ae2dcae7bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:37.163126 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0907481421c240618e1b0ae2dcae7bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:37.164476 2630 tablet_bootstrap.cc:492] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52: Bootstrap starting.
I20260928 04:53:37.164477 2632 tablet_bootstrap.cc:492] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2: Bootstrap starting.
I20260928 04:53:37.164605 2261 tablet_service.cc:1467] Processing CreateTablet for tablet 0907481421c240618e1b0ae2dcae7bf3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=96f5cec808c4467d9b1123c97ad32578]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:53:37.164752 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0907481421c240618e1b0ae2dcae7bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:37.164875 2632 tablet_bootstrap.cc:654] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:37.165186 2630 tablet_bootstrap.cc:654] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:37.165623 2632 tablet_bootstrap.cc:492] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2: No bootstrap required, opened a new log
I20260928 04:53:37.165684 2632 ts_tablet_manager.cc:1397] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:37.165827 2632 raft_consensus.cc:348] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.165875 2630 tablet_bootstrap.cc:492] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52: No bootstrap required, opened a new log
I20260928 04:53:37.165920 2632 raft_consensus.cc:374] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:37.165942 2630 ts_tablet_manager.cc:1397] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 04:53:37.165952 2632 raft_consensus.cc:729] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c6d4980832e4ce699d9261b17a700a2, State: Initialized, Role: FOLLOWER
I20260928 04:53:37.165993 2632 consensus_queue.cc:260] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.166087 2632 ts_tablet_manager.cc:1428] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:53:37.166082 2630 raft_consensus.cc:348] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.166180 2630 raft_consensus.cc:374] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:37.166209 2630 raft_consensus.cc:729] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a288da0acaa54759aaf79bbaab0e0a52, State: Initialized, Role: FOLLOWER
I20260928 04:53:37.166217 2638 tablet_replica.cc:439] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } } }
I20260928 04:53:37.166255 2630 consensus_queue.cc:260] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.166296 2638 tablet_replica.cc:442] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:53:37.166342 2630 ts_tablet_manager.cc:1428] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:37.166366 2631 tablet_bootstrap.cc:492] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e: Bootstrap starting.
I20260928 04:53:37.166448 2636 tablet_replica.cc:439] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } } }
I20260928 04:53:37.166556 2636 tablet_replica.cc:442] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:53:37.166751 2631 tablet_bootstrap.cc:654] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:37.167368 2631 tablet_bootstrap.cc:492] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e: No bootstrap required, opened a new log
I20260928 04:53:37.167415 2631 ts_tablet_manager.cc:1397] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 04:53:37.167537 2631 raft_consensus.cc:348] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.167586 2631 raft_consensus.cc:374] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:37.167599 2631 raft_consensus.cc:729] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0efa87f59a4e3bb494c0a505d6bc6e, State: Initialized, Role: FOLLOWER
I20260928 04:53:37.167652 2631 consensus_queue.cc:260] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.167780 2631 ts_tablet_manager.cc:1428] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:37.167851 2637 tablet_replica.cc:439] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } } }
I20260928 04:53:37.167960 2637 tablet_replica.cc:442] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 04:53:37.246207 2608 tablet.cc:2369] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:53:37.248024 2636 raft_consensus.cc:482] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:53:37.248142 2636 raft_consensus.cc:504] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.248528 2636 leader_election.cc:288] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b0efa87f59a4e3bb494c0a505d6bc6e (127.0.98.1:43407), 6c6d4980832e4ce699d9261b17a700a2 (127.0.98.3:44813)
I20260928 04:53:37.251813 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.248820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45353 (local address 127.0.98.1:43407)
0928 04:53:37.248873 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:37.248875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:37.248903 (+ 28us) server_negotiation.cc:407] Connection header received
0928 04:53:37.248947 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:37.248948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:37.248960 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:37.249036 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:37.249213 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.249713 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.251003 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.251406 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.251505 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.251515 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:37.251687 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 04:53:37.251727 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 04:53:37.251852 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.248850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58693 (local address 127.0.98.3:44813)
0928 04:53:37.248892 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:37.248895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:37.249023 (+ 128us) server_negotiation.cc:407] Connection header received
0928 04:53:37.249082 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:37.249083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:37.249093 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:37.249169 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:37.249323 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.249779 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.250987 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.251395 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.251496 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.251508 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:37.251719 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 04:53:37.251779 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:53:37.251868 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.248733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43407 (local address 127.0.98.2:45353)
0928 04:53:37.248790 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:37.248803 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:37.248850 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:37.249056 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:37.249057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:37.249060 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:37.249136 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.249138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.249807 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.249808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.250904 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.250907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.251508 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.251509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.251604 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.251607 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:53:37.251681 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 04:53:37.251727 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":9}
I20260928 04:53:37.251878 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.248759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44813 (local address 127.0.98.2:58693)
0928 04:53:37.248944 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:37.248961 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:37.249025 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:37.249172 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:37.249173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:37.249175 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:37.249249 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.249251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.249875 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.249877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.250682 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.250685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.251520 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.251521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.251612 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.251616 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:53:37.251708 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 04:53:37.251755 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":8,"thread_start_us":110,"threads_started":1}
I20260928 04:53:37.252163 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0907481421c240618e1b0ae2dcae7bf3" candidate_uuid: "a288da0acaa54759aaf79bbaab0e0a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" is_pre_election: true
I20260928 04:53:37.252247 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0907481421c240618e1b0ae2dcae7bf3" candidate_uuid: "a288da0acaa54759aaf79bbaab0e0a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c6d4980832e4ce699d9261b17a700a2" is_pre_election: true
I20260928 04:53:37.252331 2281 raft_consensus.cc:2454] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a288da0acaa54759aaf79bbaab0e0a52 in term 0.
I20260928 04:53:37.252355 2561 raft_consensus.cc:2454] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a288da0acaa54759aaf79bbaab0e0a52 in term 0.
I20260928 04:53:37.252542 2354 leader_election.cc:302] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0efa87f59a4e3bb494c0a505d6bc6e, a288da0acaa54759aaf79bbaab0e0a52; no voters:
I20260928 04:53:37.252699 2636 raft_consensus.cc:2781] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:53:37.252744 2636 raft_consensus.cc:482] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:53:37.252766 2636 raft_consensus.cc:3037] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:37.253451 2636 raft_consensus.cc:504] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.253611 2636 leader_election.cc:288] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [CANDIDATE]: Term 1 election: Requested vote from peers 4b0efa87f59a4e3bb494c0a505d6bc6e (127.0.98.1:43407), 6c6d4980832e4ce699d9261b17a700a2 (127.0.98.3:44813)
I20260928 04:53:37.253798 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0907481421c240618e1b0ae2dcae7bf3" candidate_uuid: "a288da0acaa54759aaf79bbaab0e0a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c6d4980832e4ce699d9261b17a700a2"
I20260928 04:53:37.253782 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0907481421c240618e1b0ae2dcae7bf3" candidate_uuid: "a288da0acaa54759aaf79bbaab0e0a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e"
I20260928 04:53:37.253875 2281 raft_consensus.cc:3037] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:37.253875 2561 raft_consensus.cc:3037] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:37.254752 2281 raft_consensus.cc:2454] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a288da0acaa54759aaf79bbaab0e0a52 in term 1.
I20260928 04:53:37.254771 2561 raft_consensus.cc:2454] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a288da0acaa54759aaf79bbaab0e0a52 in term 1.
I20260928 04:53:37.254967 2354 leader_election.cc:302] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4b0efa87f59a4e3bb494c0a505d6bc6e, 6c6d4980832e4ce699d9261b17a700a2, a288da0acaa54759aaf79bbaab0e0a52; no voters:
I20260928 04:53:37.255080 2636 raft_consensus.cc:2781] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:37.255227 2636 raft_consensus.cc:686] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [term 1 LEADER]: Becoming Leader. State: Replica: a288da0acaa54759aaf79bbaab0e0a52, State: Running, Role: LEADER
I20260928 04:53:37.255342 2636 consensus_queue.cc:237] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.255926 2652 tablet_replica.cc:439] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52: TxnStatusTablet state changed. Reason: New leader a288da0acaa54759aaf79bbaab0e0a52. Latest consensus state: current_term: 1 leader_uuid: "a288da0acaa54759aaf79bbaab0e0a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } } }
I20260928 04:53:37.256033 2652 tablet_replica.cc:442] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:53:37.256225 2654 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 04:53:37.256346 2654 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 04:53:37.256280 2124 catalog_manager.cc:5495] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 reported cstate change: term changed from 0 to 1, leader changed from <none> to a288da0acaa54759aaf79bbaab0e0a52 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a288da0acaa54759aaf79bbaab0e0a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } health_report { overall_health: UNKNOWN } } }
I20260928 04:53:37.262600 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.259777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40642 (local address 127.0.98.2:39893)
0928 04:53:37.259849 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:37.259852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:37.259954 (+ 102us) server_negotiation.cc:407] Connection header received
0928 04:53:37.260063 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:37.260064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:37.260081 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:37.260166 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:53:37.260294 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.260823 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.261669 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.261941 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.262012 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.262218 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:53:37.262329 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:53:37.262427 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:37.262470 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 04:53:37.262519 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 04:53:37.262614 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.259684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39893 (local address 127.0.0.1:40642)
0928 04:53:37.259793 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:37.259889 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:37.259979 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:37.260170 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:37.260171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:37.260173 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:53:37.260228 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.260231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.260912 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.260913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.261570 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.261573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.262013 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.262014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.262084 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.262108 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:53:37.262402 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:53:37.262413 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 04:53:37.262463 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 04:53:37.262543 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 04:53:37.264726 2561 raft_consensus.cc:1261] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2 [term 1 FOLLOWER]: Refusing update from remote peer a288da0acaa54759aaf79bbaab0e0a52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:37.264739 2281 raft_consensus.cc:1261] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 1 FOLLOWER]: Refusing update from remote peer a288da0acaa54759aaf79bbaab0e0a52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:37.265070 2636 consensus_queue.cc:1035] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:37.265208 2652 consensus_queue.cc:1035] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:37.266479 2638 tablet_replica.cc:439] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2: TxnStatusTablet state changed. Reason: New leader a288da0acaa54759aaf79bbaab0e0a52. Latest consensus state: current_term: 1 leader_uuid: "a288da0acaa54759aaf79bbaab0e0a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } } }
I20260928 04:53:37.266479 2637 tablet_replica.cc:439] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e: TxnStatusTablet state changed. Reason: New leader a288da0acaa54759aaf79bbaab0e0a52. Latest consensus state: current_term: 1 leader_uuid: "a288da0acaa54759aaf79bbaab0e0a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } } }
I20260928 04:53:37.266608 2638 tablet_replica.cc:442] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:53:37.266607 2637 tablet_replica.cc:442] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:53:37.268262 2638 tablet_replica.cc:439] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a288da0acaa54759aaf79bbaab0e0a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } } }
I20260928 04:53:37.268250 2637 tablet_replica.cc:439] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a288da0acaa54759aaf79bbaab0e0a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } } }
I20260928 04:53:37.268357 2637 tablet_replica.cc:442] T 0907481421c240618e1b0ae2dcae7bf3 P 4b0efa87f59a4e3bb494c0a505d6bc6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:53:37.268356 2638 tablet_replica.cc:442] T 0907481421c240618e1b0ae2dcae7bf3 P 6c6d4980832e4ce699d9261b17a700a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:53:37.269019 2652 tablet_replica.cc:439] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a288da0acaa54759aaf79bbaab0e0a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } } }
I20260928 04:53:37.269127 2652 tablet_replica.cc:442] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:53:37.269155 2658 tablet_replica.cc:439] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a288da0acaa54759aaf79bbaab0e0a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } } }
I20260928 04:53:37.269251 2658 tablet_replica.cc:442] T 0907481421c240618e1b0ae2dcae7bf3 P a288da0acaa54759aaf79bbaab0e0a52: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:53:37.269240 2660 mvcc.cc:205] Tried to move back new op lower bound from 7334179705913884672 to 7334179705878749184. Current Snapshot: MvccSnapshot[applied={T|T < 7334179705913884672}]
I20260928 04:53:37.273478 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.270616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.0.98.3:44813)
0928 04:53:37.270655 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:37.270657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:37.270667 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:53:37.270720 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:37.270721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:37.270736 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:37.270796 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:53:37.270977 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.271445 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.272675 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.272904 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.272974 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.273152 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:53:37.273256 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:53:37.273340 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:37.273360 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 04:53:37.273381 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:53:37.279242 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.276469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42821 (local address 127.0.98.2:39893)
0928 04:53:37.276546 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:37.276547 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:37.276557 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:53:37.276581 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:37.276581 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:37.276591 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:37.276649 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:53:37.276823 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.277297 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.278470 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.278731 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.278790 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.279019 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:53:37.279047 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:53:37.279087 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:37.279165 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 04:53:37.279182 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 04:53:37.279292 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.276243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39893 (local address 127.0.98.3:42821)
0928 04:53:37.276325 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:37.276338 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:37.276383 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:37.276668 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:37.276668 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:37.276670 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:53:37.276734 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.276735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.277374 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.277376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.278377 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.278379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.278824 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.278825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.278897 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.278915 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:53:37.279107 (+ 192us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:53:37.279109 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:53:37.279163 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 04:53:37.279206 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 04:53:37.281864 2281 raft_consensus.cc:1261] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 1 FOLLOWER]: Refusing update from remote peer 6c6d4980832e4ce699d9261b17a700a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:37.281965 2421 raft_consensus.cc:1261] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52 [term 1 FOLLOWER]: Refusing update from remote peer 6c6d4980832e4ce699d9261b17a700a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:37.282186 2638 consensus_queue.cc:1035] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:37.282349 2640 consensus_queue.cc:1035] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:37.283708 2659 mvcc.cc:205] Tried to move back new op lower bound from 7334179705984847872 to 7334179705354354688. Current Snapshot: MvccSnapshot[applied={T|T < 7334179705984847872}]
I20260928 04:53:37.292577 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.289371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44813 (local address 127.0.98.2:53839)
0928 04:53:37.289463 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:37.289477 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:37.289521 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:37.289670 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:37.289671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:37.289672 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:53:37.289735 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.289737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.290467 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.290468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.291381 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.291384 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.292025 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.292025 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.292103 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.292132 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:53:37.292381 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:53:37.292383 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:53:37.292435 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 04:53:37.292484 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 04:53:37.292564 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.289478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53839 (local address 127.0.98.3:44813)
0928 04:53:37.289526 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:37.289529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:37.289536 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:53:37.289580 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:37.289581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:37.289590 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:37.289652 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:53:37.289833 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.290359 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.291465 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.291744 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.291812 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.292252 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:53:37.292293 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:53:37.292372 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:37.292437 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 04:53:37.292463 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:53:37.303385 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:53:37.303527 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:53:37.304476 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31: Sending DeleteTablet for 3 replicas of tablet 1c82544b516c465eba26c91197b6238d
I20260928 04:53:37.304862 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c82544b516c465eba26c91197b6238d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50802
I20260928 04:53:37.305114 2676 tablet_replica.cc:330] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e: stopping tablet replica
I20260928 04:53:37.305222 2676 raft_consensus.cc:2229] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:53:37.305217 2386 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c82544b516c465eba26c91197b6238d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40636
I20260928 04:53:37.305282 2676 raft_consensus.cc:2258] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:37.305697 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c82544b516c465eba26c91197b6238d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41720
I20260928 04:53:37.305763 2676 ts_tablet_manager.cc:1905] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:37.306360 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:53:37.306565 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:53:37.307144 2677 tablet_replica.cc:330] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52: stopping tablet replica
I20260928 04:53:37.307310 2678 tablet_replica.cc:330] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2: stopping tablet replica
I20260928 04:53:37.307467 2678 raft_consensus.cc:2229] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:53:37.307636 2677 raft_consensus.cc:2229] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:53:37.307828 2678 raft_consensus.cc:2258] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:37.307947 2677 raft_consensus.cc:2258] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:37.308441 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 13532ac7b3484b98afb3ca15e2280aba (DEFAULT_TABLE table=test-table [id=ec8fbda4bf9e482c858aea91dde46e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:37.308481 2261 tablet_service.cc:1467] Processing CreateTablet for tablet 13532ac7b3484b98afb3ca15e2280aba (DEFAULT_TABLE table=test-table [id=ec8fbda4bf9e482c858aea91dde46e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:37.308564 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13532ac7b3484b98afb3ca15e2280aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:37.308595 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13532ac7b3484b98afb3ca15e2280aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:37.309901 2631 tablet_bootstrap.cc:492] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e: Bootstrap starting.
I20260928 04:53:37.309994 2386 tablet_service.cc:1467] Processing CreateTablet for tablet 13532ac7b3484b98afb3ca15e2280aba (DEFAULT_TABLE table=test-table [id=ec8fbda4bf9e482c858aea91dde46e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:37.310111 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13532ac7b3484b98afb3ca15e2280aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:37.310307 2631 tablet_bootstrap.cc:654] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:37.310607 2632 tablet_bootstrap.cc:492] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2: Bootstrap starting.
I20260928 04:53:37.311008 2632 tablet_bootstrap.cc:654] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:37.311075 2676 ts_tablet_manager.cc:1918] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:53:37.311144 2676 log.cc:1196] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/wal/wals/1c82544b516c465eba26c91197b6238d
I20260928 04:53:37.311405 2676 ts_tablet_manager.cc:1939] T 1c82544b516c465eba26c91197b6238d P 4b0efa87f59a4e3bb494c0a505d6bc6e: Deleting consensus metadata
I20260928 04:53:37.312762 2110 catalog_manager.cc:4841] TS 4b0efa87f59a4e3bb494c0a505d6bc6e (127.0.98.1:43407): tablet 1c82544b516c465eba26c91197b6238d (table test-table [id=a3df18bbd7694d5abbf7fce8a2690cd3]) successfully deleted
I20260928 04:53:37.313215 2630 tablet_bootstrap.cc:492] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52: Bootstrap starting.
I20260928 04:53:37.313648 2630 tablet_bootstrap.cc:654] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:37.314298 2630 tablet_bootstrap.cc:492] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52: No bootstrap required, opened a new log
I20260928 04:53:37.314354 2630 ts_tablet_manager.cc:1397] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 04:53:37.314541 2630 raft_consensus.cc:348] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.314622 2630 raft_consensus.cc:374] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:37.314648 2630 raft_consensus.cc:729] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a288da0acaa54759aaf79bbaab0e0a52, State: Initialized, Role: FOLLOWER
I20260928 04:53:37.314689 2630 consensus_queue.cc:260] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.314782 2630 ts_tablet_manager.cc:1428] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:53:37.315039 2678 ts_tablet_manager.cc:1905] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:37.315131 2677 ts_tablet_manager.cc:1905] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:37.317087 2677 ts_tablet_manager.cc:1918] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:53:37.317173 2677 log.cc:1196] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/1c82544b516c465eba26c91197b6238d
I20260928 04:53:37.317442 2677 ts_tablet_manager.cc:1939] T 1c82544b516c465eba26c91197b6238d P a288da0acaa54759aaf79bbaab0e0a52: Deleting consensus metadata
I20260928 04:53:37.318094 2109 catalog_manager.cc:4841] TS a288da0acaa54759aaf79bbaab0e0a52 (127.0.98.2:39893): tablet 1c82544b516c465eba26c91197b6238d (table test-table [id=a3df18bbd7694d5abbf7fce8a2690cd3]) successfully deleted
I20260928 04:53:37.318905 2631 tablet_bootstrap.cc:492] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e: No bootstrap required, opened a new log
I20260928 04:53:37.318928 2632 tablet_bootstrap.cc:492] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2: No bootstrap required, opened a new log
I20260928 04:53:37.318986 2632 ts_tablet_manager.cc:1397] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260928 04:53:37.318988 2631 ts_tablet_manager.cc:1397] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260928 04:53:37.319149 2632 raft_consensus.cc:348] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.319232 2632 raft_consensus.cc:374] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:37.319178 2631 raft_consensus.cc:348] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.319255 2632 raft_consensus.cc:729] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c6d4980832e4ce699d9261b17a700a2, State: Initialized, Role: FOLLOWER
I20260928 04:53:37.319267 2631 raft_consensus.cc:374] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:37.319291 2631 raft_consensus.cc:729] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0efa87f59a4e3bb494c0a505d6bc6e, State: Initialized, Role: FOLLOWER
I20260928 04:53:37.319316 2632 consensus_queue.cc:260] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.319358 2631 consensus_queue.cc:260] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.319404 2632 ts_tablet_manager.cc:1428] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:53:37.319463 2631 ts_tablet_manager.cc:1428] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:37.320294 2678 ts_tablet_manager.cc:1918] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:53:37.320381 2678 log.cc:1196] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/1c82544b516c465eba26c91197b6238d
I20260928 04:53:37.320716 2678 ts_tablet_manager.cc:1939] T 1c82544b516c465eba26c91197b6238d P 6c6d4980832e4ce699d9261b17a700a2: Deleting consensus metadata
I20260928 04:53:37.321137 2109 catalog_manager.cc:4841] TS 6c6d4980832e4ce699d9261b17a700a2 (127.0.98.3:44813): tablet 1c82544b516c465eba26c91197b6238d (table test-table [id=a3df18bbd7694d5abbf7fce8a2690cd3]) successfully deleted
I20260928 04:53:37.452401 2664 raft_consensus.cc:482] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:53:37.452517 2664 raft_consensus.cc:504] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.452733 2664 leader_election.cc:288] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b0efa87f59a4e3bb494c0a505d6bc6e (127.0.98.1:43407), 6c6d4980832e4ce699d9261b17a700a2 (127.0.98.3:44813)
I20260928 04:53:37.452970 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13532ac7b3484b98afb3ca15e2280aba" candidate_uuid: "a288da0acaa54759aaf79bbaab0e0a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c6d4980832e4ce699d9261b17a700a2" is_pre_election: true
I20260928 04:53:37.453039 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13532ac7b3484b98afb3ca15e2280aba" candidate_uuid: "a288da0acaa54759aaf79bbaab0e0a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" is_pre_election: true
I20260928 04:53:37.453091 2561 raft_consensus.cc:2454] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a288da0acaa54759aaf79bbaab0e0a52 in term 0.
I20260928 04:53:37.453121 2281 raft_consensus.cc:2454] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a288da0acaa54759aaf79bbaab0e0a52 in term 0.
I20260928 04:53:37.453315 2353 leader_election.cc:302] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c6d4980832e4ce699d9261b17a700a2, a288da0acaa54759aaf79bbaab0e0a52; no voters:
I20260928 04:53:37.453437 2664 raft_consensus.cc:2781] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:53:37.453509 2664 raft_consensus.cc:482] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:53:37.453538 2664 raft_consensus.cc:3037] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:37.454257 2664 raft_consensus.cc:504] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.454413 2664 leader_election.cc:288] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [CANDIDATE]: Term 1 election: Requested vote from peers 4b0efa87f59a4e3bb494c0a505d6bc6e (127.0.98.1:43407), 6c6d4980832e4ce699d9261b17a700a2 (127.0.98.3:44813)
I20260928 04:53:37.454583 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13532ac7b3484b98afb3ca15e2280aba" candidate_uuid: "a288da0acaa54759aaf79bbaab0e0a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e"
I20260928 04:53:37.454669 2281 raft_consensus.cc:3037] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:37.454628 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13532ac7b3484b98afb3ca15e2280aba" candidate_uuid: "a288da0acaa54759aaf79bbaab0e0a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c6d4980832e4ce699d9261b17a700a2"
I20260928 04:53:37.454756 2561 raft_consensus.cc:3037] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:37.455463 2281 raft_consensus.cc:2454] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a288da0acaa54759aaf79bbaab0e0a52 in term 1.
I20260928 04:53:37.455621 2354 leader_election.cc:302] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0efa87f59a4e3bb494c0a505d6bc6e, a288da0acaa54759aaf79bbaab0e0a52; no voters:
I20260928 04:53:37.455632 2561 raft_consensus.cc:2454] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a288da0acaa54759aaf79bbaab0e0a52 in term 1.
I20260928 04:53:37.455763 2664 raft_consensus.cc:2781] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:37.455828 2664 raft_consensus.cc:686] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [term 1 LEADER]: Becoming Leader. State: Replica: a288da0acaa54759aaf79bbaab0e0a52, State: Running, Role: LEADER
I20260928 04:53:37.455904 2664 consensus_queue.cc:237] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:37.456537 2124 catalog_manager.cc:5495] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 reported cstate change: term changed from 0 to 1, leader changed from <none> to a288da0acaa54759aaf79bbaab0e0a52 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a288da0acaa54759aaf79bbaab0e0a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } health_report { overall_health: UNKNOWN } } }
I20260928 04:53:37.497991 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.495629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.0.98.2:39893)
0928 04:53:37.495692 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:37.495694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:37.495703 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:53:37.495764 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:37.495765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:37.495775 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:37.495830 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:53:37.496013 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.496559 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.497313 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.497526 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.497604 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.497746 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:53:37.497779 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:53:37.497826 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:37.497884 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 04:53:37.497902 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:53:37.502830 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.500454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39893 (local address 127.0.98.2:55241)
0928 04:53:37.500509 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:37.500521 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:37.500600 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:37.500730 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:37.500730 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:37.500731 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:53:37.500767 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.500768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.501523 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.501523 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.502063 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:37.502065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.502427 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.502427 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:37.502462 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.502467 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:53:37.502659 (+ 192us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:53:37.502660 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 04:53:37.502691 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 04:53:37.502727 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 04:53:37.502833 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:37.500520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55241 (local address 127.0.98.2:39893)
0928 04:53:37.500575 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:37.500577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:37.500587 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:53:37.500668 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:37.500669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:37.500676 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:37.500726 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:53:37.500892 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.501451 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.502150 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:37.502363 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:37.502418 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:37.502585 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:53:37.502609 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:53:37.502649 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:37.502710 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 04:53:37.502729 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:53:37.504813 2561 raft_consensus.cc:1261] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2 [term 1 FOLLOWER]: Refusing update from remote peer a288da0acaa54759aaf79bbaab0e0a52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:37.504899 2281 raft_consensus.cc:1261] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 1 FOLLOWER]: Refusing update from remote peer a288da0acaa54759aaf79bbaab0e0a52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:37.505045 2658 consensus_queue.cc:1035] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:37.505139 2652 consensus_queue.cc:1035] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:37.506361 2657 mvcc.cc:205] Tried to move back new op lower bound from 7334179706898563072 to 7334179706700058624. Current Snapshot: MvccSnapshot[applied={T|T < 7334179706898563072}]
W20260928 04:53:37.509999 2657 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 04:53:39.029486 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:53:39.029588 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:53:39.030457 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31: Sending DeleteTablet for 3 replicas of tablet 13532ac7b3484b98afb3ca15e2280aba
I20260928 04:53:39.030745 2386 tablet_service.cc:1514] Processing DeleteTablet for tablet 13532ac7b3484b98afb3ca15e2280aba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40636
I20260928 04:53:39.030772 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 13532ac7b3484b98afb3ca15e2280aba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41720
I20260928 04:53:39.030826 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet 13532ac7b3484b98afb3ca15e2280aba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50802
I20260928 04:53:39.031015 2708 tablet_replica.cc:330] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e: stopping tablet replica
I20260928 04:53:39.031103 2708 raft_consensus.cc:2229] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:53:39.031152 2708 raft_consensus.cc:2258] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:39.031396 2708 ts_tablet_manager.cc:1905] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:39.033289 2708 ts_tablet_manager.cc:1918] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 04:53:39.033375 2708 log.cc:1196] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/wal/wals/13532ac7b3484b98afb3ca15e2280aba
I20260928 04:53:39.033633 2708 ts_tablet_manager.cc:1939] T 13532ac7b3484b98afb3ca15e2280aba P 4b0efa87f59a4e3bb494c0a505d6bc6e: Deleting consensus metadata
I20260928 04:53:39.033772 2706 tablet_replica.cc:330] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52: stopping tablet replica
I20260928 04:53:39.033842 2706 raft_consensus.cc:2229] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:53:39.033986 2706 raft_consensus.cc:2258] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:39.034039 2110 catalog_manager.cc:4841] TS 4b0efa87f59a4e3bb494c0a505d6bc6e (127.0.98.1:43407): tablet 13532ac7b3484b98afb3ca15e2280aba (table test-table [id=ec8fbda4bf9e482c858aea91dde46e2a]) successfully deleted
I20260928 04:53:39.034126 2707 tablet_replica.cc:330] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2: stopping tablet replica
I20260928 04:53:39.034198 2707 raft_consensus.cc:2229] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:53:39.034251 2707 raft_consensus.cc:2258] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:39.034354 2706 ts_tablet_manager.cc:1905] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:39.034571 2707 ts_tablet_manager.cc:1905] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:39.036132 2707 ts_tablet_manager.cc:1918] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 04:53:39.036207 2707 log.cc:1196] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/13532ac7b3484b98afb3ca15e2280aba
I20260928 04:53:39.036423 2707 ts_tablet_manager.cc:1939] T 13532ac7b3484b98afb3ca15e2280aba P 6c6d4980832e4ce699d9261b17a700a2: Deleting consensus metadata
I20260928 04:53:39.036604 2706 ts_tablet_manager.cc:1918] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 04:53:39.036662 2706 log.cc:1196] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/13532ac7b3484b98afb3ca15e2280aba
I20260928 04:53:39.036767 2109 catalog_manager.cc:4841] TS 6c6d4980832e4ce699d9261b17a700a2 (127.0.98.3:44813): tablet 13532ac7b3484b98afb3ca15e2280aba (table test-table [id=ec8fbda4bf9e482c858aea91dde46e2a]) successfully deleted
I20260928 04:53:39.036911 2706 ts_tablet_manager.cc:1939] T 13532ac7b3484b98afb3ca15e2280aba P a288da0acaa54759aaf79bbaab0e0a52: Deleting consensus metadata
I20260928 04:53:39.037348 2109 catalog_manager.cc:4841] TS a288da0acaa54759aaf79bbaab0e0a52 (127.0.98.2:39893): tablet 13532ac7b3484b98afb3ca15e2280aba (table test-table [id=ec8fbda4bf9e482c858aea91dde46e2a]) successfully deleted
Sep 28 04:53:39 dist-test-slave-m39l krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571219, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 04:53:39 dist-test-slave-m39l krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571219, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:39.056427 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:39.048785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.0.98.62:33749)
0928 04:53:39.048985 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:39.048987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:39.048996 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:53:39.049024 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:39.049024 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:39.049035 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:39.049116 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:39.049701 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.050253 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.051130 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.051375 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.051442 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:39.053903 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:39.053905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:39.053905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:39.053913 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:39.055361 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:39.055689 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:39.055690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:39.055690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:39.055743 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:39.055978 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:39.055979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:39.055979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:39.056094 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:39.056167 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:39.056253 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 04:53:39.056319 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":99,"threads_started":1}
I20260928 04:53:39.058389 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:53:39.058756 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:53:39.060305 2261 tablet_service.cc:1467] Processing CreateTablet for tablet 7bafd2aa348144f4aa4a56d80716e5c0 (DEFAULT_TABLE table=test-table [id=fdf81bfa65174e7b8cbe21aa74a28944]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:39.060379 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 7bafd2aa348144f4aa4a56d80716e5c0 (DEFAULT_TABLE table=test-table [id=fdf81bfa65174e7b8cbe21aa74a28944]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:39.060467 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bafd2aa348144f4aa4a56d80716e5c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:39.060534 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bafd2aa348144f4aa4a56d80716e5c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:39.061687 2717 tablet_bootstrap.cc:492] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e: Bootstrap starting.
I20260928 04:53:39.061750 2386 tablet_service.cc:1467] Processing CreateTablet for tablet 7bafd2aa348144f4aa4a56d80716e5c0 (DEFAULT_TABLE table=test-table [id=fdf81bfa65174e7b8cbe21aa74a28944]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:39.061866 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bafd2aa348144f4aa4a56d80716e5c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:39.061702 2718 tablet_bootstrap.cc:492] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2: Bootstrap starting.
I20260928 04:53:39.062130 2717 tablet_bootstrap.cc:654] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:39.062461 2718 tablet_bootstrap.cc:654] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:39.062927 2717 tablet_bootstrap.cc:492] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e: No bootstrap required, opened a new log
I20260928 04:53:39.062983 2717 ts_tablet_manager.cc:1397] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:39.063135 2717 raft_consensus.cc:348] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:39.063225 2717 raft_consensus.cc:374] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:39.063201 2718 tablet_bootstrap.cc:492] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2: No bootstrap required, opened a new log
I20260928 04:53:39.063253 2717 raft_consensus.cc:729] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0efa87f59a4e3bb494c0a505d6bc6e, State: Initialized, Role: FOLLOWER
I20260928 04:53:39.063279 2718 ts_tablet_manager.cc:1397] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:53:39.063298 2717 consensus_queue.cc:260] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:39.063395 2718 raft_consensus.cc:348] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:39.063475 2718 raft_consensus.cc:374] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:39.063506 2718 raft_consensus.cc:729] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c6d4980832e4ce699d9261b17a700a2, State: Initialized, Role: FOLLOWER
I20260928 04:53:39.063581 2718 consensus_queue.cc:260] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:39.063609 2717 ts_tablet_manager.cc:1428] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:39.063618 2721 tablet_bootstrap.cc:492] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52: Bootstrap starting.
I20260928 04:53:39.063798 2718 ts_tablet_manager.cc:1428] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:53:39.064002 2721 tablet_bootstrap.cc:654] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:39.064539 2721 tablet_bootstrap.cc:492] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52: No bootstrap required, opened a new log
I20260928 04:53:39.064597 2721 ts_tablet_manager.cc:1397] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:39.064705 2721 raft_consensus.cc:348] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:39.064796 2721 raft_consensus.cc:374] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:39.064900 2721 raft_consensus.cc:729] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a288da0acaa54759aaf79bbaab0e0a52, State: Initialized, Role: FOLLOWER
I20260928 04:53:39.064980 2721 consensus_queue.cc:260] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:39.065116 2721 ts_tablet_manager.cc:1428] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:53:39.101647 2701 raft_consensus.cc:482] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:53:39.101818 2701 raft_consensus.cc:504] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:39.102007 2701 leader_election.cc:288] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b0efa87f59a4e3bb494c0a505d6bc6e (127.0.98.1:43407), 6c6d4980832e4ce699d9261b17a700a2 (127.0.98.3:44813)
I20260928 04:53:39.102252 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bafd2aa348144f4aa4a56d80716e5c0" candidate_uuid: "a288da0acaa54759aaf79bbaab0e0a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" is_pre_election: true
I20260928 04:53:39.102294 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bafd2aa348144f4aa4a56d80716e5c0" candidate_uuid: "a288da0acaa54759aaf79bbaab0e0a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c6d4980832e4ce699d9261b17a700a2" is_pre_election: true
I20260928 04:53:39.102373 2561 raft_consensus.cc:2454] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a288da0acaa54759aaf79bbaab0e0a52 in term 0.
I20260928 04:53:39.102370 2281 raft_consensus.cc:2454] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a288da0acaa54759aaf79bbaab0e0a52 in term 0.
I20260928 04:53:39.102568 2353 leader_election.cc:302] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c6d4980832e4ce699d9261b17a700a2, a288da0acaa54759aaf79bbaab0e0a52; no voters:
I20260928 04:53:39.102710 2701 raft_consensus.cc:2781] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:53:39.102779 2701 raft_consensus.cc:482] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:53:39.102811 2701 raft_consensus.cc:3037] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:39.103524 2701 raft_consensus.cc:504] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:39.103682 2701 leader_election.cc:288] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [CANDIDATE]: Term 1 election: Requested vote from peers 4b0efa87f59a4e3bb494c0a505d6bc6e (127.0.98.1:43407), 6c6d4980832e4ce699d9261b17a700a2 (127.0.98.3:44813)
I20260928 04:53:39.103852 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bafd2aa348144f4aa4a56d80716e5c0" candidate_uuid: "a288da0acaa54759aaf79bbaab0e0a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e"
I20260928 04:53:39.103876 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bafd2aa348144f4aa4a56d80716e5c0" candidate_uuid: "a288da0acaa54759aaf79bbaab0e0a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c6d4980832e4ce699d9261b17a700a2"
I20260928 04:53:39.103960 2281 raft_consensus.cc:3037] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:39.103966 2561 raft_consensus.cc:3037] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:39.104554 2281 raft_consensus.cc:2454] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a288da0acaa54759aaf79bbaab0e0a52 in term 1.
I20260928 04:53:39.104761 2354 leader_election.cc:302] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0efa87f59a4e3bb494c0a505d6bc6e, a288da0acaa54759aaf79bbaab0e0a52; no voters:
I20260928 04:53:39.104800 2561 raft_consensus.cc:2454] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a288da0acaa54759aaf79bbaab0e0a52 in term 1.
I20260928 04:53:39.104868 2701 raft_consensus.cc:2781] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:39.104918 2701 raft_consensus.cc:686] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [term 1 LEADER]: Becoming Leader. State: Replica: a288da0acaa54759aaf79bbaab0e0a52, State: Running, Role: LEADER
I20260928 04:53:39.104991 2701 consensus_queue.cc:237] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:39.105712 2123 catalog_manager.cc:5495] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 reported cstate change: term changed from 0 to 1, leader changed from <none> to a288da0acaa54759aaf79bbaab0e0a52 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a288da0acaa54759aaf79bbaab0e0a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } health_report { overall_health: UNKNOWN } } }
I20260928 04:53:39.118799 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:39.116197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40672 (local address 127.0.98.2:39893)
0928 04:53:39.116350 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:39.116352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:39.116360 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:53:39.116389 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:39.116389 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:39.116399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:39.116462 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:53:39.116627 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.117135 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.117902 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.118113 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.118185 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:39.118325 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:53:39.118354 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:53:39.118398 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:39.118578 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 04:53:39.118610 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":106,"threads_started":1}
I20260928 04:53:39.121821 2281 raft_consensus.cc:1261] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 1 FOLLOWER]: Refusing update from remote peer a288da0acaa54759aaf79bbaab0e0a52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:39.121833 2561 raft_consensus.cc:1261] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2 [term 1 FOLLOWER]: Refusing update from remote peer a288da0acaa54759aaf79bbaab0e0a52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:39.122077 2636 consensus_queue.cc:1035] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:39.122169 2701 consensus_queue.cc:1035] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:39.133364 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:53:39.133507 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:53:39.134213 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31: Sending DeleteTablet for 3 replicas of tablet 7bafd2aa348144f4aa4a56d80716e5c0
I20260928 04:53:39.134467 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bafd2aa348144f4aa4a56d80716e5c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50802
I20260928 04:53:39.134577 2386 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bafd2aa348144f4aa4a56d80716e5c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40636
I20260928 04:53:39.134614 2708 tablet_replica.cc:330] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e: stopping tablet replica
I20260928 04:53:39.134630 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bafd2aa348144f4aa4a56d80716e5c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41720
I20260928 04:53:39.134665 2708 raft_consensus.cc:2229] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:53:39.134728 2706 tablet_replica.cc:330] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52: stopping tablet replica
I20260928 04:53:39.134744 2708 raft_consensus.cc:2258] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:39.134795 2706 raft_consensus.cc:2229] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:53:39.134860 2706 raft_consensus.cc:2258] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:39.134951 2707 tablet_replica.cc:330] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2: stopping tablet replica
I20260928 04:53:39.135016 2707 raft_consensus.cc:2229] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:53:39.135054 2707 raft_consensus.cc:2258] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:39.135025 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 04:53:39.135116 2708 ts_tablet_manager.cc:1905] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 04:53:39.135175 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:53:39.135363 2707 ts_tablet_manager.cc:1905] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:39.135417 2706 ts_tablet_manager.cc:1905] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:39.136734 2708 ts_tablet_manager.cc:1918] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:53:39.136816 2708 log.cc:1196] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-0/wal/wals/7bafd2aa348144f4aa4a56d80716e5c0
I20260928 04:53:39.137066 2707 ts_tablet_manager.cc:1918] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:53:39.137120 2708 ts_tablet_manager.cc:1939] T 7bafd2aa348144f4aa4a56d80716e5c0 P 4b0efa87f59a4e3bb494c0a505d6bc6e: Deleting consensus metadata
I20260928 04:53:39.137127 2707 log.cc:1196] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/7bafd2aa348144f4aa4a56d80716e5c0
I20260928 04:53:39.137342 2707 ts_tablet_manager.cc:1939] T 7bafd2aa348144f4aa4a56d80716e5c0 P 6c6d4980832e4ce699d9261b17a700a2: Deleting consensus metadata
I20260928 04:53:39.137413 2706 ts_tablet_manager.cc:1918] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:53:39.137476 2706 log.cc:1196] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/7bafd2aa348144f4aa4a56d80716e5c0
I20260928 04:53:39.137552 2110 catalog_manager.cc:4841] TS 4b0efa87f59a4e3bb494c0a505d6bc6e (127.0.98.1:43407): tablet 7bafd2aa348144f4aa4a56d80716e5c0 (table test-table [id=fdf81bfa65174e7b8cbe21aa74a28944]) successfully deleted
I20260928 04:53:39.137715 2706 ts_tablet_manager.cc:1939] T 7bafd2aa348144f4aa4a56d80716e5c0 P a288da0acaa54759aaf79bbaab0e0a52: Deleting consensus metadata
I20260928 04:53:39.137794 2109 catalog_manager.cc:4841] TS 6c6d4980832e4ce699d9261b17a700a2 (127.0.98.3:44813): tablet 7bafd2aa348144f4aa4a56d80716e5c0 (table test-table [id=fdf81bfa65174e7b8cbe21aa74a28944]) successfully deleted
I20260928 04:53:39.138164 2109 catalog_manager.cc:4841] TS a288da0acaa54759aaf79bbaab0e0a52 (127.0.98.2:39893): tablet 7bafd2aa348144f4aa4a56d80716e5c0 (table test-table [id=fdf81bfa65174e7b8cbe21aa74a28944]) successfully deleted
I20260928 04:53:39.138217 2261 tablet_service.cc:1467] Processing CreateTablet for tablet ad3691fe09e0492abc445716fad87472 (DEFAULT_TABLE table=test-table [id=a0c5844f352a480d99db35057851899a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:39.138260 2386 tablet_service.cc:1467] Processing CreateTablet for tablet ad3691fe09e0492abc445716fad87472 (DEFAULT_TABLE table=test-table [id=a0c5844f352a480d99db35057851899a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:39.138319 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3691fe09e0492abc445716fad87472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:39.138325 2538 tablet_service.cc:1467] Processing CreateTablet for tablet ad3691fe09e0492abc445716fad87472 (DEFAULT_TABLE table=test-table [id=a0c5844f352a480d99db35057851899a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:39.138357 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3691fe09e0492abc445716fad87472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:39.138410 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3691fe09e0492abc445716fad87472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:39.139488 2721 tablet_bootstrap.cc:492] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52: Bootstrap starting.
I20260928 04:53:39.139582 2718 tablet_bootstrap.cc:492] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2: Bootstrap starting.
I20260928 04:53:39.139806 2721 tablet_bootstrap.cc:654] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:39.139955 2718 tablet_bootstrap.cc:654] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:39.140223 2717 tablet_bootstrap.cc:492] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e: Bootstrap starting.
I20260928 04:53:39.140401 2721 tablet_bootstrap.cc:492] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52: No bootstrap required, opened a new log
I20260928 04:53:39.140462 2721 ts_tablet_manager.cc:1397] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:39.140487 2718 tablet_bootstrap.cc:492] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2: No bootstrap required, opened a new log
I20260928 04:53:39.140532 2718 ts_tablet_manager.cc:1397] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:39.140576 2721 raft_consensus.cc:348] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:39.140661 2721 raft_consensus.cc:374] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:39.140686 2721 raft_consensus.cc:729] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a288da0acaa54759aaf79bbaab0e0a52, State: Initialized, Role: FOLLOWER
I20260928 04:53:39.140663 2718 raft_consensus.cc:348] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:39.140726 2718 raft_consensus.cc:374] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:39.140756 2718 raft_consensus.cc:729] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c6d4980832e4ce699d9261b17a700a2, State: Initialized, Role: FOLLOWER
I20260928 04:53:39.140731 2721 consensus_queue.cc:260] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:39.140796 2718 consensus_queue.cc:260] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:39.140828 2721 ts_tablet_manager.cc:1428] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:53:39.140893 2718 ts_tablet_manager.cc:1428] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:39.141004 2717 tablet_bootstrap.cc:654] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:39.141824 2717 tablet_bootstrap.cc:492] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e: No bootstrap required, opened a new log
I20260928 04:53:39.141902 2717 ts_tablet_manager.cc:1397] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:53:39.142048 2717 raft_consensus.cc:348] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:39.142128 2717 raft_consensus.cc:374] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:39.142153 2717 raft_consensus.cc:729] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0efa87f59a4e3bb494c0a505d6bc6e, State: Initialized, Role: FOLLOWER
I20260928 04:53:39.142202 2717 consensus_queue.cc:260] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:39.142299 2717 ts_tablet_manager.cc:1428] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:39.217901 2722 raft_consensus.cc:482] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:53:39.218010 2722 raft_consensus.cc:504] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:39.218385 2722 leader_election.cc:288] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a288da0acaa54759aaf79bbaab0e0a52 (127.0.98.2:39893), 6c6d4980832e4ce699d9261b17a700a2 (127.0.98.3:44813)
I20260928 04:53:39.222644 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:39.218632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39893 (local address 127.0.98.1:33025)
0928 04:53:39.218772 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:39.218785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:39.218842 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:39.219064 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:39.219065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:39.219068 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:39.219177 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:39.219179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.220221 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.220222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:39.221654 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:39.221657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.222323 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.222325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:39.222406 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:39.222410 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:53:39.222474 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 04:53:39.222544 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":66,"thread_start_us":93,"threads_started":1}
I20260928 04:53:39.222622 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:39.218786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33025 (local address 127.0.98.2:39893)
0928 04:53:39.218900 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:39.218903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:39.218915 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:53:39.218951 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:39.218953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:39.218964 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:39.219045 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:39.219279 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.220061 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.221820 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.222234 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.222302 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:39.222312 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:39.222508 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 04:53:39.222552 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 04:53:39.222851 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:39.218925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39419 (local address 127.0.98.3:44813)
0928 04:53:39.219037 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:39.219038 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:39.219048 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:53:39.219133 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:39.219134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:39.219142 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:39.219204 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:39.219403 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.220107 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.221781 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.222337 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.222409 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:39.222422 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:39.222662 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 04:53:39.222715 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
I20260928 04:53:39.222793 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:39.218840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44813 (local address 127.0.98.1:39419)
0928 04:53:39.218988 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:39.218996 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:39.219061 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:39.219228 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:39.219228 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:39.219231 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:39.219302 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:39.219303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.220190 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.220191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:39.221654 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:39.221689 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.222454 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.222455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:39.222583 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:39.222587 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:53:39.222654 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 04:53:39.222697 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":89,"thread_start_us":69,"threads_started":1}
I20260928 04:53:39.222995 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3691fe09e0492abc445716fad87472" candidate_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a288da0acaa54759aaf79bbaab0e0a52" is_pre_election: true
I20260928 04:53:39.223091 2421 raft_consensus.cc:2454] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0efa87f59a4e3bb494c0a505d6bc6e in term 0.
I20260928 04:53:39.223114 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3691fe09e0492abc445716fad87472" candidate_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c6d4980832e4ce699d9261b17a700a2" is_pre_election: true
I20260928 04:53:39.223191 2561 raft_consensus.cc:2454] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0efa87f59a4e3bb494c0a505d6bc6e in term 0.
I20260928 04:53:39.223351 2213 leader_election.cc:302] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0efa87f59a4e3bb494c0a505d6bc6e, a288da0acaa54759aaf79bbaab0e0a52; no voters:
I20260928 04:53:39.223541 2722 raft_consensus.cc:2781] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:53:39.223590 2722 raft_consensus.cc:482] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:53:39.223604 2722 raft_consensus.cc:3037] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:39.224330 2722 raft_consensus.cc:504] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:39.224484 2722 leader_election.cc:288] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [CANDIDATE]: Term 1 election: Requested vote from peers a288da0acaa54759aaf79bbaab0e0a52 (127.0.98.2:39893), 6c6d4980832e4ce699d9261b17a700a2 (127.0.98.3:44813)
I20260928 04:53:39.224661 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3691fe09e0492abc445716fad87472" candidate_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a288da0acaa54759aaf79bbaab0e0a52"
I20260928 04:53:39.224718 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3691fe09e0492abc445716fad87472" candidate_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c6d4980832e4ce699d9261b17a700a2"
I20260928 04:53:39.224766 2421 raft_consensus.cc:3037] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:39.224802 2561 raft_consensus.cc:3037] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:39.225394 2561 raft_consensus.cc:2454] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b0efa87f59a4e3bb494c0a505d6bc6e in term 1.
I20260928 04:53:39.225483 2421 raft_consensus.cc:2454] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b0efa87f59a4e3bb494c0a505d6bc6e in term 1.
I20260928 04:53:39.225561 2213 leader_election.cc:302] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0efa87f59a4e3bb494c0a505d6bc6e, 6c6d4980832e4ce699d9261b17a700a2; no voters:
I20260928 04:53:39.225693 2722 raft_consensus.cc:2781] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:39.225891 2722 raft_consensus.cc:686] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [term 1 LEADER]: Becoming Leader. State: Replica: 4b0efa87f59a4e3bb494c0a505d6bc6e, State: Running, Role: LEADER
I20260928 04:53:39.225970 2722 consensus_queue.cc:237] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } }
I20260928 04:53:39.226735 2124 catalog_manager.cc:5495] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b0efa87f59a4e3bb494c0a505d6bc6e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0efa87f59a4e3bb494c0a505d6bc6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 } health_report { overall_health: UNKNOWN } } }
I20260928 04:53:39.237758 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:39.235414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.0.98.1:43407)
0928 04:53:39.235602 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:39.235604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:39.235611 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:53:39.235645 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:39.235646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:39.235665 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:39.235735 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:53:39.235831 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.236368 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.237054 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.237253 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.237308 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:39.237417 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:53:39.237522 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:53:39.237572 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:39.237607 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 04:53:39.237623 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":126,"threads_started":1}
I20260928 04:53:39.243090 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:39.240417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39893 (local address 127.0.98.1:58195)
0928 04:53:39.240613 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:39.240624 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:39.240710 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:39.240851 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:39.240852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:39.240854 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:53:39.240924 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:39.240925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.241553 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.241554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:39.242162 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:39.242164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.242620 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.242622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:39.242677 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:39.242691 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:53:39.242895 (+ 204us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:53:39.242897 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:53:39.242958 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 04:53:39.242997 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":98,"threads_started":1}
I20260928 04:53:39.243070 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:39.240519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58195 (local address 127.0.98.2:39893)
0928 04:53:39.240590 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:39.240592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:39.240694 (+ 102us) server_negotiation.cc:407] Connection header received
0928 04:53:39.240762 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:39.240763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:39.240775 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:39.240837 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:53:39.240993 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.241472 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.242256 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.242462 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.242592 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:39.242790 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:53:39.242834 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:53:39.242892 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:39.242962 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 04:53:39.242985 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 04:53:39.245533 2421 raft_consensus.cc:1261] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52 [term 1 FOLLOWER]: Refusing update from remote peer 4b0efa87f59a4e3bb494c0a505d6bc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:39.245533 2561 raft_consensus.cc:1261] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2 [term 1 FOLLOWER]: Refusing update from remote peer 4b0efa87f59a4e3bb494c0a505d6bc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:39.245803 2722 consensus_queue.cc:1035] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c6d4980832e4ce699d9261b17a700a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:39.245947 2737 consensus_queue.cc:1035] T ad3691fe09e0492abc445716fad87472 P 4b0efa87f59a4e3bb494c0a505d6bc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a288da0acaa54759aaf79bbaab0e0a52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:39.253474 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:39.250955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60507 (local address 127.0.98.1:43407)
0928 04:53:39.251000 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:39.251002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:39.251031 (+ 29us) server_negotiation.cc:407] Connection header received
0928 04:53:39.251128 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:39.251129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:39.251142 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:39.251193 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:53:39.251371 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.251837 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.252769 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.253077 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.253145 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:39.253289 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:53:39.253315 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:53:39.253347 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:39.253395 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 04:53:39.253412 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:53:39.253490 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:39.250826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43407 (local address 127.0.98.2:60507)
0928 04:53:39.250973 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:39.250985 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:39.251045 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:39.251236 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:39.251237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:39.251239 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:53:39.251313 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:39.251315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.251908 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.251909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:39.252684 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:39.252687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:39.253154 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:39.253155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:39.253215 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:39.253221 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:53:39.253363 (+ 142us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:53:39.253364 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 04:53:39.253394 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 04:53:39.253432 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
W20260928 04:53:39.253913 2692 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 04:53:40.778322 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:53:40.778421 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:53:40.779179 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55b5ad84fe754720a4a51dcbb2079b31: Sending DeleteTablet for 3 replicas of tablet ad3691fe09e0492abc445716fad87472
I20260928 04:53:40.779440 2386 tablet_service.cc:1514] Processing DeleteTablet for tablet ad3691fe09e0492abc445716fad87472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40636
I20260928 04:53:40.779459 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet ad3691fe09e0492abc445716fad87472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41720
I20260928 04:53:40.779608 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 2196
I20260928 04:53:40.779654 2772 tablet_replica.cc:330] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2: stopping tablet replica
I20260928 04:53:40.779709 2772 raft_consensus.cc:2229] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:53:40.780257 2771 tablet_replica.cc:330] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52: stopping tablet replica
I20260928 04:53:40.780323 2771 raft_consensus.cc:2229] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:53:40.780364 2771 raft_consensus.cc:2258] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:40.780534 2772 raft_consensus.cc:2258] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:40.780596 2771 ts_tablet_manager.cc:1905] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:40.780815 2772 ts_tablet_manager.cc:1905] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:40.782451 2772 ts_tablet_manager.cc:1918] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 04:53:40.782585 2772 log.cc:1196] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/ad3691fe09e0492abc445716fad87472
I20260928 04:53:40.782606 2771 ts_tablet_manager.cc:1918] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 04:53:40.782665 2771 log.cc:1196] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/ad3691fe09e0492abc445716fad87472
I20260928 04:53:40.782896 2771 ts_tablet_manager.cc:1939] T ad3691fe09e0492abc445716fad87472 P a288da0acaa54759aaf79bbaab0e0a52: Deleting consensus metadata
I20260928 04:53:40.782907 2772 ts_tablet_manager.cc:1939] T ad3691fe09e0492abc445716fad87472 P 6c6d4980832e4ce699d9261b17a700a2: Deleting consensus metadata
I20260928 04:53:40.783303 2109 catalog_manager.cc:4841] TS 6c6d4980832e4ce699d9261b17a700a2 (127.0.98.3:44813): tablet ad3691fe09e0492abc445716fad87472 (table test-table [id=a0c5844f352a480d99db35057851899a]) successfully deleted
I20260928 04:53:40.783396 2109 catalog_manager.cc:4841] TS a288da0acaa54759aaf79bbaab0e0a52 (127.0.98.2:39893): tablet ad3691fe09e0492abc445716fad87472 (table test-table [id=a0c5844f352a480d99db35057851899a]) successfully deleted
W20260928 04:53:40.787679 2110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43407 (error 108)
I20260928 04:53:40.787910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 2336
I20260928 04:53:40.789019 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:40.788718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43407 (local address 127.0.0.1:50820)
0928 04:53:40.788864 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:40.788962 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
W20260928 04:53:40.789193 2110 catalog_manager.cc:4584] TS 4b0efa87f59a4e3bb494c0a505d6bc6e (127.0.98.1:43407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad3691fe09e0492abc445716fad87472: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43407: connect: Connection refused (error 111)
I20260928 04:53:40.794464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 2476
I20260928 04:53:40.800616 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 2092
2026-09-28T04:53:40Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5729 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:53:40 dist-test-slave-m39l krb5kdc[2777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:53:40 dist-test-slave-m39l krb5kdc[2777](info): set up 2 sockets
Sep 28 04:53:40 dist-test-slave-m39l krb5kdc[2777](info): commencing operation
krb5kdc: starting...
W20260928 04:53:41.906395 392 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:53:41 dist-test-slave-m39l krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:53:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:53:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:53:42.034770 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37429
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:42.117152 2790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:42.117383 2790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:42.117408 2790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:42.118589 2790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:53:42.118631 2790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:42.118646 2790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:42.118659 2790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:53:42.118671 2790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:53:42.120194 2790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:42.120416 2790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:42.120601 2790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:53:42.125933 2790 server_base.cc:1034] running on GCE node
W20260928 04:53:42.125948 2798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:42.126133 2795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:42.125933 2796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:42.126543 2790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:42.126843 2790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:42.127996 2790 hybrid_clock.cc:648] HybridClock initialized: now 1790571222127984 us; error 31 us; skew 500 ppm
Sep 28 04:53:42 dist-test-slave-m39l krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:42.134456 2790 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:53:42.135100 2790 webserver.cc:460] Webserver started at http://127.0.98.62:34827/ using document root <none> and password file <none>
I20260928 04:53:42.135304 2790 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:42.135346 2790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:42.135440 2790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:42.136507 2790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "85b0ddf8ddb24ee68ed08beed7347164"
format_stamp: "Formatted at 2026-09-28 04:53:42 on dist-test-slave-m39l"
server_key: "03ad2dfbba4f5c6349fef031981af213"
server_key_iv: "70af2197a334e678ac468be57bddfe1c"
server_key_version: "encryptionkey@0"
I20260928 04:53:42.136809 2790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "85b0ddf8ddb24ee68ed08beed7347164"
format_stamp: "Formatted at 2026-09-28 04:53:42 on dist-test-slave-m39l"
server_key: "03ad2dfbba4f5c6349fef031981af213"
server_key_iv: "70af2197a334e678ac468be57bddfe1c"
server_key_version: "encryptionkey@0"
I20260928 04:53:42.138134 2790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:53:42.138998 2805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:42.139225 2790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:53:42.139329 2790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "85b0ddf8ddb24ee68ed08beed7347164"
format_stamp: "Formatted at 2026-09-28 04:53:42 on dist-test-slave-m39l"
server_key: "03ad2dfbba4f5c6349fef031981af213"
server_key_iv: "70af2197a334e678ac468be57bddfe1c"
server_key_version: "encryptionkey@0"
I20260928 04:53:42.139406 2790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:42.149180 2790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:42.151678 2790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:42.151839 2790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:42.156291 2790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37429
I20260928 04:53:42.156327 2857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37429 every 8 connection(s)
I20260928 04:53:42.156876 2790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:53:42.157608 2858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:42.159992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 2790
I20260928 04:53:42.160095 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:53:42.160292 392 external_mini_cluster.cc:1428] Setting key 298707d19065764963d4da1bb230d839
I20260928 04:53:42.160641 2858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164: Bootstrap starting.
I20260928 04:53:42.161397 2858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:42.161736 2858 log.cc:824] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:42.162468 2858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164: No bootstrap required, opened a new log
I20260928 04:53:42.164502 2858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b0ddf8ddb24ee68ed08beed7347164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37429 } }
I20260928 04:53:42.164677 2858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:42.164759 2858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b0ddf8ddb24ee68ed08beed7347164, State: Initialized, Role: FOLLOWER
I20260928 04:53:42.164963 2858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b0ddf8ddb24ee68ed08beed7347164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37429 } }
I20260928 04:53:42.165105 2858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:53:42.165150 2858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:53:42.165215 2858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:42.165846 2858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b0ddf8ddb24ee68ed08beed7347164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37429 } }
I20260928 04:53:42.165978 2858 leader_election.cc:302] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85b0ddf8ddb24ee68ed08beed7347164; no voters:
I20260928 04:53:42.166190 2858 leader_election.cc:288] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:53:42.166227 2863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:42.166343 2863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164 [term 1 LEADER]: Becoming Leader. State: Replica: 85b0ddf8ddb24ee68ed08beed7347164, State: Running, Role: LEADER
I20260928 04:53:42.166440 2863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b0ddf8ddb24ee68ed08beed7347164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37429 } }
I20260928 04:53:42.166519 2858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:42.166903 2863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85b0ddf8ddb24ee68ed08beed7347164. Latest consensus state: current_term: 1 leader_uuid: "85b0ddf8ddb24ee68ed08beed7347164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b0ddf8ddb24ee68ed08beed7347164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37429 } } }
I20260928 04:53:42.166914 2865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85b0ddf8ddb24ee68ed08beed7347164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b0ddf8ddb24ee68ed08beed7347164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37429 } } }
I20260928 04:53:42.166980 2863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:42.167013 2865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164 [sys.catalog]: This master's current role is: LEADER
Sep 28 04:53:42 dist-test-slave-m39l krb5kdc[2777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:42.167511 2868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:53:42.168124 2868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:53:42.169953 2868 catalog_manager.cc:1347] Generated new cluster ID: fc050a4a712f488c9f655c92dccdaa0d
I20260928 04:53:42.170002 2868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:53:42.171476 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:42.160889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50354 (local address 127.0.98.62:37429)
0928 04:53:42.161093 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:42.161101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:42.161129 (+ 28us) server_negotiation.cc:407] Connection header received
0928 04:53:42.161646 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:42.161653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:42.161703 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:42.161861 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:42.162582 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:42.163499 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:42.164274 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:42.164678 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:42.164847 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:42.167661 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:42.167665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:42.167665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:42.167678 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:42.169681 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:42.170097 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:42.170100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:42.170101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:42.170213 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:42.170441 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:42.170442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:42.170442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:42.170822 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:42.170997 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:42.171081 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 04:53:42.171239 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":93,"threads_started":1}
I20260928 04:53:42.189206 2868 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:53:42.189986 2868 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:53:42.195801 2868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85b0ddf8ddb24ee68ed08beed7347164: Generated new TSK 0
I20260928 04:53:42.196004 2868 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:53:42.236490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37429
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:42.322808 2884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:42.323062 2884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:42.323107 2884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:42.324357 2884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:42.324438 2884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:42.324491 2884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:53:42.326128 2884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:42.326424 2884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:42.326713 2884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:42.332114 2892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:42.332114 2890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:42.332114 2889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:42.332629 2884 server_base.cc:1034] running on GCE node
I20260928 04:53:42.332829 2884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:42.333046 2884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:42.334187 2884 hybrid_clock.cc:648] HybridClock initialized: now 1790571222334172 us; error 30 us; skew 500 ppm
Sep 28 04:53:42 dist-test-slave-m39l krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:42.336927 2884 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:53:42.337457 2884 webserver.cc:460] Webserver started at http://127.0.98.1:44341/ using document root <none> and password file <none>
I20260928 04:53:42.337705 2884 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:42.337760 2884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:42.337927 2884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:42.339159 2884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "3c70c7726672429198a55e68e14d4ac7"
format_stamp: "Formatted at 2026-09-28 04:53:42 on dist-test-slave-m39l"
server_key: "834474718634bead004b7dd2f41ecacf"
server_key_iv: "02e083065968a9d06977bd05c4e847fd"
server_key_version: "encryptionkey@0"
I20260928 04:53:42.339569 2884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3c70c7726672429198a55e68e14d4ac7"
format_stamp: "Formatted at 2026-09-28 04:53:42 on dist-test-slave-m39l"
server_key: "834474718634bead004b7dd2f41ecacf"
server_key_iv: "02e083065968a9d06977bd05c4e847fd"
server_key_version: "encryptionkey@0"
I20260928 04:53:42.341045 2884 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:53:42.342072 2899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:42.342329 2884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:42.342427 2884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "3c70c7726672429198a55e68e14d4ac7"
format_stamp: "Formatted at 2026-09-28 04:53:42 on dist-test-slave-m39l"
server_key: "834474718634bead004b7dd2f41ecacf"
server_key_iv: "02e083065968a9d06977bd05c4e847fd"
server_key_version: "encryptionkey@0"
I20260928 04:53:42.342531 2884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:42.353955 2884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:42.356426 2884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:42.356609 2884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:42.356877 2884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:42.357205 2884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:42.357241 2884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:42.357286 2884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:42.357312 2884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:42.364315 2884 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42439
I20260928 04:53:42.364367 3012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42439 every 8 connection(s)
I20260928 04:53:42.364765 2884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:53:42.371915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 2884
Sep 28 04:53:42 dist-test-slave-m39l krb5kdc[2777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:42.372058 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:53:42.372282 392 external_mini_cluster.cc:1428] Setting key a96e5e5bac1e94872a6157f8de34e0e5
I20260928 04:53:42.376075 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:42.365708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57121 (local address 127.0.98.62:37429)
0928 04:53:42.365825 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:42.365828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:42.366011 (+ 183us) server_negotiation.cc:407] Connection header received
0928 04:53:42.367145 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:42.367147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:42.367159 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:42.367274 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:42.368005 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:42.368652 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:42.369745 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:42.369971 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:42.370034 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:42.372802 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:42.372804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:42.372804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:42.372812 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:42.374299 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:42.374853 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:42.374863 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:42.374864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:42.374932 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:42.375219 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:42.375219 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:42.375220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:42.375374 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:42.375444 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:42.375806 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 04:53:42.375926 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 04:53:42.376219 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:42.365674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37429 (local address 127.0.98.1:57121)
0928 04:53:42.365872 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:42.365913 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:42.367035 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:42.367375 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:42.367379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:42.367661 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:42.367874 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:42.367879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:42.368752 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:42.368753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:42.369646 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:42.369650 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:42.370091 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:42.370092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:42.370231 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:42.370912 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:42.370922 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:42.372690 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:42.374406 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:42.374418 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:42.374421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:42.374768 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:42.375005 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:42.375006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:42.375006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:42.375143 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:42.375481 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:42.375488 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:42.375640 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 04:53:42.375747 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 04:53:42.375964 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":118,"threads_started":1}
I20260928 04:53:42.376762 3013 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37429
I20260928 04:53:42.376900 3013 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:42.377102 3013 heartbeater.cc:511] Master 127.0.98.62:37429 requested a full tablet report, sending...
I20260928 04:53:42.377700 2822 ts_manager.cc:194] Registered new tserver with Master: 3c70c7726672429198a55e68e14d4ac7 (127.0.98.1:42439)
I20260928 04:53:42.378404 2822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57121
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:53:42.401852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37429
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:42.485910 3018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:42.486213 3018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:42.486259 3018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:42.487661 3018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:42.487767 3018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:42.487814 3018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:53:42.489568 3018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:42.489897 3018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:42.490164 3018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:53:42.495708 3018 server_base.cc:1034] running on GCE node
W20260928 04:53:42.495676 3026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:42.495679 3024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:42.495680 3023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:42.496251 3018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:42.496521 3018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:42.497735 3018 hybrid_clock.cc:648] HybridClock initialized: now 1790571222497736 us; error 45 us; skew 500 ppm
Sep 28 04:53:42 dist-test-slave-m39l krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:42.500495 3018 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:53:42.500960 3018 webserver.cc:460] Webserver started at http://127.0.98.2:46841/ using document root <none> and password file <none>
I20260928 04:53:42.501199 3018 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:42.501276 3018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:42.501426 3018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:42.502609 3018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "b52c75d9a2a1458d9a5f98a819b23e8d"
format_stamp: "Formatted at 2026-09-28 04:53:42 on dist-test-slave-m39l"
server_key: "50098c92d8b302bea8f6934b697064d9"
server_key_iv: "057b164bf8e6aef2785d4e9845286602"
server_key_version: "encryptionkey@0"
I20260928 04:53:42.502979 3018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b52c75d9a2a1458d9a5f98a819b23e8d"
format_stamp: "Formatted at 2026-09-28 04:53:42 on dist-test-slave-m39l"
server_key: "50098c92d8b302bea8f6934b697064d9"
server_key_iv: "057b164bf8e6aef2785d4e9845286602"
server_key_version: "encryptionkey@0"
I20260928 04:53:42.504369 3018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:53:42.505087 3033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:42.505295 3018 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:53:42.505404 3018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "b52c75d9a2a1458d9a5f98a819b23e8d"
format_stamp: "Formatted at 2026-09-28 04:53:42 on dist-test-slave-m39l"
server_key: "50098c92d8b302bea8f6934b697064d9"
server_key_iv: "057b164bf8e6aef2785d4e9845286602"
server_key_version: "encryptionkey@0"
I20260928 04:53:42.505479 3018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:42.520687 3018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:42.523376 3018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:42.523562 3018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:42.523818 3018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:42.524166 3018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:42.524227 3018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:42.524273 3018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:42.524320 3018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:42.530258 3018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39101
I20260928 04:53:42.530295 3146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39101 every 8 connection(s)
I20260928 04:53:42.530714 3018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:53:42 dist-test-slave-m39l krb5kdc[2777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:42.537616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 3018
I20260928 04:53:42.537753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:53:42.538003 392 external_mini_cluster.cc:1428] Setting key 7a23a6b8f299289482dcb961435a4ef3
I20260928 04:53:42.541410 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:42.531548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49355 (local address 127.0.98.62:37429)
0928 04:53:42.531619 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:42.531621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:42.531860 (+ 239us) server_negotiation.cc:407] Connection header received
0928 04:53:42.532829 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:42.532830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:42.532841 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:42.532904 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:42.533492 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:42.534009 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:42.535107 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:42.535293 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:42.535347 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:42.537799 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:42.537801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:42.537801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:42.537811 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:42.539654 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:42.540140 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:42.540141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:42.540141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:42.540215 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:42.540695 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:42.540697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:42.540697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:42.540836 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:42.540919 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:42.541214 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 04:53:42.541313 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:53:42.541637 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:42.531538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37429 (local address 127.0.98.2:49355)
0928 04:53:42.531725 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:42.531762 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:42.532719 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:42.532961 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:42.532964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:42.533219 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:42.533421 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:42.533425 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:42.534092 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:42.534093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:42.535034 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:42.535037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:42.535365 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:42.535366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:42.535473 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:42.536054 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:42.536062 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:42.537635 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:42.539776 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:42.539785 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:42.539788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:42.540051 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:42.540467 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:42.540468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:42.540469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:42.540611 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:42.540909 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:42.540916 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:42.541066 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 04:53:42.541191 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 04:53:42.541373 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260928 04:53:42.542282 3147 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37429
I20260928 04:53:42.542398 3147 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:42.542637 3147 heartbeater.cc:511] Master 127.0.98.62:37429 requested a full tablet report, sending...
I20260928 04:53:42.543102 2822 ts_manager.cc:194] Registered new tserver with Master: b52c75d9a2a1458d9a5f98a819b23e8d (127.0.98.2:39101)
I20260928 04:53:42.543659 2822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49355
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:53:42.566533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37429
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:42.649722 3152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:42.649977 3152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:42.650005 3152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:42.651257 3152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:42.651309 3152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:42.651336 3152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:53:42.652891 3152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:42.653162 3152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:42.653414 3152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:53:42.658641 3152 server_base.cc:1034] running on GCE node
W20260928 04:53:42.658638 3158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:42.658821 3157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:42.658663 3160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:42.659142 3152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:42.659409 3152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:42.660583 3152 hybrid_clock.cc:648] HybridClock initialized: now 1790571222660559 us; error 40 us; skew 500 ppm
Sep 28 04:53:42 dist-test-slave-m39l krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:42.663429 3152 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:53:42.663956 3152 webserver.cc:460] Webserver started at http://127.0.98.3:39987/ using document root <none> and password file <none>
I20260928 04:53:42.664207 3152 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:42.664285 3152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:42.664441 3152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:42.665542 3152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "da0eeb4602af479b81775852f8dc57ad"
format_stamp: "Formatted at 2026-09-28 04:53:42 on dist-test-slave-m39l"
server_key: "290dd257ebcf74c9e26c38e03ab19339"
server_key_iv: "45c3a6f484f1f236887ad1f13e271892"
server_key_version: "encryptionkey@0"
I20260928 04:53:42.665915 3152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "da0eeb4602af479b81775852f8dc57ad"
format_stamp: "Formatted at 2026-09-28 04:53:42 on dist-test-slave-m39l"
server_key: "290dd257ebcf74c9e26c38e03ab19339"
server_key_iv: "45c3a6f484f1f236887ad1f13e271892"
server_key_version: "encryptionkey@0"
I20260928 04:53:42.667516 3152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:53:42.668555 3167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:42.668710 3152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:53:42.668797 3152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "da0eeb4602af479b81775852f8dc57ad"
format_stamp: "Formatted at 2026-09-28 04:53:42 on dist-test-slave-m39l"
server_key: "290dd257ebcf74c9e26c38e03ab19339"
server_key_iv: "45c3a6f484f1f236887ad1f13e271892"
server_key_version: "encryptionkey@0"
I20260928 04:53:42.668905 3152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:42.686959 3152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:42.689503 3152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:42.689667 3152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:42.689955 3152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:42.690287 3152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:42.690321 3152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:42.690366 3152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:42.690392 3152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:42.696722 3152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46439
I20260928 04:53:42.696832 3280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46439 every 8 connection(s)
I20260928 04:53:42.697145 3152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:53:42.702097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 3152
I20260928 04:53:42.702219 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:53:42.702430 392 external_mini_cluster.cc:1428] Setting key 0327f87dc1e55ee3c84612ca109bb913
Sep 28 04:53:42 dist-test-slave-m39l krb5kdc[2777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:42.707958 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:42.697971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44625 (local address 127.0.98.62:37429)
0928 04:53:42.698065 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:42.698067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:42.698282 (+ 215us) server_negotiation.cc:407] Connection header received
0928 04:53:42.699604 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:42.699606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:42.699621 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:42.699673 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:42.700341 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:42.700861 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:42.701966 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:42.702174 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:42.702262 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:42.705040 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:42.705042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:42.705042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:42.705051 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:42.706376 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:42.706867 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:42.706868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:42.706869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:42.706936 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:42.707230 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:42.707231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:42.707231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:42.707357 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:42.707438 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:42.707767 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 04:53:42.707853 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 04:53:42.708179 3283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:42.697888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37429 (local address 127.0.98.3:44625)
0928 04:53:42.698105 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:42.698185 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:42.699462 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:42.699760 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:42.699764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:42.700042 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:42.700239 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:42.700245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:42.700960 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:42.700961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:42.701812 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:42.701814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:42.702262 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:42.702263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:42.702403 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:42.703092 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:42.703105 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:42.704916 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:42.706474 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:42.706482 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:42.706485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:42.706778 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:42.707014 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:42.707015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:42.707015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:42.707146 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:42.707443 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:42.707450 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:42.707608 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 04:53:42.707731 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 04:53:42.707916 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260928 04:53:42.708772 3281 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37429
I20260928 04:53:42.708878 3281 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:42.709028 3281 heartbeater.cc:511] Master 127.0.98.62:37429 requested a full tablet report, sending...
I20260928 04:53:42.709439 2822 ts_manager.cc:194] Registered new tserver with Master: da0eeb4602af479b81775852f8dc57ad (127.0.98.3:46439)
I20260928 04:53:42.709913 2822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44625
I20260928 04:53:42.715066 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 04:53:42.727690 2861 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:53:42.726430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50358 (local address 127.0.98.62:37429)
0928 04:53:42.726548 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:42.726550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:42.726734 (+ 184us) server_negotiation.cc:407] Connection header received
0928 04:53:42.726861 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:42.726862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:42.726875 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:42.726943 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:42.727574 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50358: BlockingRecv error: recv got EOF from 127.0.0.1:50358 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 04:53:42.729135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 2884
I20260928 04:53:42.733762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 3018
I20260928 04:53:42.738379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 3152
I20260928 04:53:42.742885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 2790
2026-09-28T04:53:42Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1941 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:53:42 dist-test-slave-m39l krb5kdc[3293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:53:42 dist-test-slave-m39l krb5kdc[3293](info): set up 2 sockets
Sep 28 04:53:42 dist-test-slave-m39l krb5kdc[3293](info): commencing operation
krb5kdc: starting...
W20260928 04:53:43.814564 392 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:53:43 dist-test-slave-m39l krb5kdc[3293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:53:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:53:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:53:43.943538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39345
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:44.032456 3306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:44.032799 3306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:44.032838 3306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:44.034785 3306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:53:44.034859 3306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:44.034895 3306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:44.034930 3306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:53:44.034969 3306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:53:44.037146 3306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:44.037472 3306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:44.037734 3306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:44.043041 3314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:44.043133 3306 server_base.cc:1034] running on GCE node
W20260928 04:53:44.043020 3312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:44.043020 3311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:44.043517 3306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:44.043874 3306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:44.045037 3306 hybrid_clock.cc:648] HybridClock initialized: now 1790571224045021 us; error 30 us; skew 500 ppm
Sep 28 04:53:44 dist-test-slave-m39l krb5kdc[3293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:44.047839 3306 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:53:44.048369 3306 webserver.cc:460] Webserver started at http://127.0.98.62:40811/ using document root <none> and password file <none>
I20260928 04:53:44.048632 3306 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:44.048696 3306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:44.048854 3306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:44.050001 3306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "14228d2d94424443b4ddc5812c2306e8"
format_stamp: "Formatted at 2026-09-28 04:53:44 on dist-test-slave-m39l"
server_key: "ee3485fc9a7dbbb1ddf58153e5445428"
server_key_iv: "33b49bad54fbdcea55f7a173669ec99c"
server_key_version: "encryptionkey@0"
I20260928 04:53:44.050375 3306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "14228d2d94424443b4ddc5812c2306e8"
format_stamp: "Formatted at 2026-09-28 04:53:44 on dist-test-slave-m39l"
server_key: "ee3485fc9a7dbbb1ddf58153e5445428"
server_key_iv: "33b49bad54fbdcea55f7a173669ec99c"
server_key_version: "encryptionkey@0"
I20260928 04:53:44.051887 3306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:53:44.052821 3321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:44.053077 3306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:53:44.053180 3306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "14228d2d94424443b4ddc5812c2306e8"
format_stamp: "Formatted at 2026-09-28 04:53:44 on dist-test-slave-m39l"
server_key: "ee3485fc9a7dbbb1ddf58153e5445428"
server_key_iv: "33b49bad54fbdcea55f7a173669ec99c"
server_key_version: "encryptionkey@0"
I20260928 04:53:44.053260 3306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:44.079537 3306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:44.081728 3306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:44.081902 3306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:44.085889 3306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39345
I20260928 04:53:44.085930 3373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39345 every 8 connection(s)
I20260928 04:53:44.086303 3306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:53:44.087265 3374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:44.088832 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 3306
I20260928 04:53:44.088903 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:53:44.089079 392 external_mini_cluster.cc:1428] Setting key c41eafd6b057919bf7dfab79cf6e7e02
I20260928 04:53:44.090092 3374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8: Bootstrap starting.
I20260928 04:53:44.090904 3374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:44.091240 3374 log.cc:824] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:44.091938 3374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8: No bootstrap required, opened a new log
I20260928 04:53:44.093760 3374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14228d2d94424443b4ddc5812c2306e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39345 } }
I20260928 04:53:44.093937 3374 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:44.094000 3374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14228d2d94424443b4ddc5812c2306e8, State: Initialized, Role: FOLLOWER
I20260928 04:53:44.094135 3374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14228d2d94424443b4ddc5812c2306e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39345 } }
I20260928 04:53:44.094254 3374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:53:44.094291 3374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:53:44.094352 3374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:44.095242 3374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14228d2d94424443b4ddc5812c2306e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39345 } }
I20260928 04:53:44.095376 3374 leader_election.cc:302] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14228d2d94424443b4ddc5812c2306e8; no voters:
I20260928 04:53:44.095569 3374 leader_election.cc:288] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:53:44.095628 3379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:44.095816 3379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8 [term 1 LEADER]: Becoming Leader. State: Replica: 14228d2d94424443b4ddc5812c2306e8, State: Running, Role: LEADER
I20260928 04:53:44.095868 3374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:44.095929 3379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14228d2d94424443b4ddc5812c2306e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39345 } }
Sep 28 04:53:44 dist-test-slave-m39l krb5kdc[3293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:44.096274 3381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14228d2d94424443b4ddc5812c2306e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14228d2d94424443b4ddc5812c2306e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39345 } } }
I20260928 04:53:44.096251 3379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14228d2d94424443b4ddc5812c2306e8. Latest consensus state: current_term: 1 leader_uuid: "14228d2d94424443b4ddc5812c2306e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14228d2d94424443b4ddc5812c2306e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39345 } } }
I20260928 04:53:44.096355 3381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:44.096364 3379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:44.097482 3386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:53:44.098065 3386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:53:44.101033 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:44.089652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38564 (local address 127.0.98.62:39345)
0928 04:53:44.089859 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:44.089870 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:44.089903 (+ 33us) server_negotiation.cc:407] Connection header received
0928 04:53:44.090473 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:44.090515 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:44.090592 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:44.090753 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:44.091421 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:44.092401 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:44.093158 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:44.093577 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:44.093708 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:44.097006 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:44.097012 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:44.097012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:44.097026 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:44.099220 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:44.099749 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:44.099752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:44.099752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:44.099901 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:44.100117 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:44.100117 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:44.100118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:44.100350 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:44.100462 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:44.100613 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 04:53:44.100725 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
I20260928 04:53:44.102087 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:53:44.102995 3386 catalog_manager.cc:1347] Generated new cluster ID: a9bf90910d3046a8a2735bb9cb55828a
I20260928 04:53:44.103051 3386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:53:44.120221 3386 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:53:44.120831 3386 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:53:44.123335 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:44.119038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38574 (local address 127.0.98.62:39345)
0928 04:53:44.119083 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:44.119085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:44.119231 (+ 146us) server_negotiation.cc:407] Connection header received
0928 04:53:44.119311 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:44.119312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:44.119325 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:44.119393 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:44.119860 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:44.120527 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:44.121625 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:44.121940 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:44.122005 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:44.122829 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:44.122830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:44.122830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:53:44.123054 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:44.123073 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:53:44.123079 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:44.123134 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:44.123214 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 04:53:44.123256 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:53:44.129879 3386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14228d2d94424443b4ddc5812c2306e8: Generated new TSK 0
I20260928 04:53:44.130120 3386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:53:44.134392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 3306
2026-09-28T04:53:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1389 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:53:44 dist-test-slave-m39l krb5kdc[3407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:53:44 dist-test-slave-m39l krb5kdc[3407](info): set up 2 sockets
Sep 28 04:53:44 dist-test-slave-m39l krb5kdc[3407](info): commencing operation
krb5kdc: starting...
W20260928 04:53:45.194463 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:53:45 dist-test-slave-m39l krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:53:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:53:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:53:45.398087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35263
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:45.482209 3420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:45.482445 3420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:45.482468 3420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:45.483741 3420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:53:45.483793 3420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:45.483807 3420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:45.483822 3420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:53:45.483834 3420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:53:45.485352 3420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:45.485572 3420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:45.485764 3420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:45.490818 3425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:45.490815 3428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:45.490815 3426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:45.491277 3420 server_base.cc:1034] running on GCE node
I20260928 04:53:45.491523 3420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:45.491817 3420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:45.493023 3420 hybrid_clock.cc:648] HybridClock initialized: now 1790571225493002 us; error 31 us; skew 500 ppm
Sep 28 04:53:45 dist-test-slave-m39l krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:45.496230 3420 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:53:45.496910 3420 webserver.cc:460] Webserver started at http://127.0.98.62:42377/ using document root <none> and password file <none>
I20260928 04:53:45.497179 3420 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:45.497265 3420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:45.497447 3420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:45.498800 3420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "eb85348a79164ac98e6b1e002cb53707"
format_stamp: "Formatted at 2026-09-28 04:53:45 on dist-test-slave-m39l"
server_key: "831baa1a1805b1e7934fe2e33eeaae15"
server_key_iv: "4b6dd8ee5430522cbbd76a1e31a76eba"
server_key_version: "encryptionkey@0"
I20260928 04:53:45.499238 3420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb85348a79164ac98e6b1e002cb53707"
format_stamp: "Formatted at 2026-09-28 04:53:45 on dist-test-slave-m39l"
server_key: "831baa1a1805b1e7934fe2e33eeaae15"
server_key_iv: "4b6dd8ee5430522cbbd76a1e31a76eba"
server_key_version: "encryptionkey@0"
I20260928 04:53:45.500880 3420 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:53:45.501695 3435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:45.501880 3420 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:53:45.501991 3420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "eb85348a79164ac98e6b1e002cb53707"
format_stamp: "Formatted at 2026-09-28 04:53:45 on dist-test-slave-m39l"
server_key: "831baa1a1805b1e7934fe2e33eeaae15"
server_key_iv: "4b6dd8ee5430522cbbd76a1e31a76eba"
server_key_version: "encryptionkey@0"
I20260928 04:53:45.502070 3420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:45.517036 3420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:45.519603 3420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:45.519790 3420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:45.524148 3420 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35263
I20260928 04:53:45.524194 3487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35263 every 8 connection(s)
I20260928 04:53:45.524606 3420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:53:45.525684 3488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:45.528255 3488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707: Bootstrap starting.
I20260928 04:53:45.528903 3488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:45.529217 3488 log.cc:824] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:45.530126 3488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707: No bootstrap required, opened a new log
I20260928 04:53:45.531935 3488 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb85348a79164ac98e6b1e002cb53707" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35263 } }
I20260928 04:53:45.532096 3488 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:45.532119 3488 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb85348a79164ac98e6b1e002cb53707, State: Initialized, Role: FOLLOWER
I20260928 04:53:45.532291 3488 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb85348a79164ac98e6b1e002cb53707" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35263 } }
I20260928 04:53:45.532379 3488 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:53:45.532444 3488 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:53:45.532503 3488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:45.533463 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 3420
I20260928 04:53:45.533509 3488 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb85348a79164ac98e6b1e002cb53707" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35263 } }
I20260928 04:53:45.533566 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:53:45.533622 3488 leader_election.cc:302] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb85348a79164ac98e6b1e002cb53707; no voters:
I20260928 04:53:45.533746 392 external_mini_cluster.cc:1428] Setting key a9318030322f9bcdb965c8c914c0843f
I20260928 04:53:45.533815 3488 leader_election.cc:288] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:53:45.533890 3491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:45.534081 3491 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707 [term 1 LEADER]: Becoming Leader. State: Replica: eb85348a79164ac98e6b1e002cb53707, State: Running, Role: LEADER
I20260928 04:53:45.534241 3491 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb85348a79164ac98e6b1e002cb53707" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35263 } }
I20260928 04:53:45.534093 3488 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:45.534629 3492 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb85348a79164ac98e6b1e002cb53707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb85348a79164ac98e6b1e002cb53707" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35263 } } }
I20260928 04:53:45.534725 3492 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:45.534890 3493 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb85348a79164ac98e6b1e002cb53707. Latest consensus state: current_term: 1 leader_uuid: "eb85348a79164ac98e6b1e002cb53707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb85348a79164ac98e6b1e002cb53707" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35263 } } }
I20260928 04:53:45.534955 3493 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:45.534934 3498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:53:45.535844 3498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:53:45.537799 3498 catalog_manager.cc:1347] Generated new cluster ID: 7d4d473afcf2436d9c247b307021b339
I20260928 04:53:45.537860 3498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:53:45 dist-test-slave-m39l krb5kdc[3407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:45.544831 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:45.534366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39150 (local address 127.0.98.62:35263)
0928 04:53:45.534632 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:45.534642 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:45.534678 (+ 36us) server_negotiation.cc:407] Connection header received
0928 04:53:45.534817 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:45.534826 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:45.534909 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:45.535747 (+ 838us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:45.535954 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:45.537155 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:45.538208 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:45.538721 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:45.538849 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:45.541275 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:45.541282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:45.541282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:45.541301 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:45.543150 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:45.543583 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:45.543587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:45.543588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:45.543701 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:45.543927 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:45.543928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:45.543929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:45.544203 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:45.544403 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:45.544484 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 04:53:45.544642 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":68,"threads_started":1}
I20260928 04:53:45.558418 3498 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:53:45.559067 3498 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:53:45.567533 3498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb85348a79164ac98e6b1e002cb53707: Generated new TSK 0
I20260928 04:53:45.567777 3498 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:53:45.603565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35263
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:45.693185 3514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:45.693424 3514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:45.693451 3514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:45.694764 3514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:45.694819 3514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:45.694849 3514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:53:45.696367 3514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:45.696579 3514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:45.696776 3514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:45.702078 3519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:45.702075 3520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:45.702160 3514 server_base.cc:1034] running on GCE node
W20260928 04:53:45.702075 3522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:45.702522 3514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:45.702764 3514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:45.703951 3514 hybrid_clock.cc:648] HybridClock initialized: now 1790571225703938 us; error 30 us; skew 500 ppm
Sep 28 04:53:45 dist-test-slave-m39l krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:45.706828 3514 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:53:45.707324 3514 webserver.cc:460] Webserver started at http://127.0.98.1:43505/ using document root <none> and password file <none>
I20260928 04:53:45.707587 3514 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:45.707662 3514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:45.707813 3514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:45.708900 3514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "332e81da38ba4d759944d649af0dec71"
format_stamp: "Formatted at 2026-09-28 04:53:45 on dist-test-slave-m39l"
server_key: "373fbbd905078d0450cc6aee428a762b"
server_key_iv: "cb34582df6e46d5613d5bc7c58702fda"
server_key_version: "encryptionkey@0"
I20260928 04:53:45.709334 3514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "332e81da38ba4d759944d649af0dec71"
format_stamp: "Formatted at 2026-09-28 04:53:45 on dist-test-slave-m39l"
server_key: "373fbbd905078d0450cc6aee428a762b"
server_key_iv: "cb34582df6e46d5613d5bc7c58702fda"
server_key_version: "encryptionkey@0"
I20260928 04:53:45.710872 3514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:53:45.711725 3529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:45.711900 3514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:45.712008 3514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "332e81da38ba4d759944d649af0dec71"
format_stamp: "Formatted at 2026-09-28 04:53:45 on dist-test-slave-m39l"
server_key: "373fbbd905078d0450cc6aee428a762b"
server_key_iv: "cb34582df6e46d5613d5bc7c58702fda"
server_key_version: "encryptionkey@0"
I20260928 04:53:45.712086 3514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:45.731245 3514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:45.733799 3514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:45.733985 3514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:45.734246 3514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:45.734802 3514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:45.734870 3514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:45.734916 3514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:45.734946 3514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:45.741401 3514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35827
I20260928 04:53:45.741488 3642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35827 every 8 connection(s)
I20260928 04:53:45.741849 3514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:53:45 dist-test-slave-m39l krb5kdc[3407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:45.749771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 3514
I20260928 04:53:45.749893 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:53:45.750094 392 external_mini_cluster.cc:1428] Setting key 1d1591f32f2da72e7ae640c468a05c01
I20260928 04:53:45.753309 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:45.742869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52235 (local address 127.0.98.62:35263)
0928 04:53:45.742924 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:45.742927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:45.742960 (+ 33us) server_negotiation.cc:407] Connection header received
0928 04:53:45.743550 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:45.743552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:45.743569 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:45.743616 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:45.744625 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:45.745142 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:45.746169 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:45.746379 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:45.746457 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:45.748766 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:45.748768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:45.748768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:45.748775 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:45.750384 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:45.752132 (+ 1748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:45.752133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:45.752133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:45.752195 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:45.752549 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:45.752549 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:45.752550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:45.752659 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:45.752712 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:45.753086 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 04:53:45.753214 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:53:45.753468 3645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:45.742657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35263 (local address 127.0.98.1:52235)
0928 04:53:45.742835 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:45.742882 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:45.743428 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:45.743694 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:45.743698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:45.744309 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:45.744519 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:45.744524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:45.745242 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:45.745243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:45.746074 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:45.746075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:45.746465 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:45.746466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:45.746633 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:45.747242 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:45.747250 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:45.748681 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:45.750445 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:45.750454 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:45.750457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:45.750818 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:45.752280 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:45.752281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:45.752281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:45.752473 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:45.752723 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:45.752729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:45.752926 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 04:53:45.753067 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 04:53:45.753234 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
I20260928 04:53:45.754053 3643 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35263
I20260928 04:53:45.754169 3643 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:45.754354 3643 heartbeater.cc:511] Master 127.0.98.62:35263 requested a full tablet report, sending...
I20260928 04:53:45.754915 3452 ts_manager.cc:194] Registered new tserver with Master: 332e81da38ba4d759944d649af0dec71 (127.0.98.1:35827)
I20260928 04:53:45.755564 3452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52235
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:53:45.781852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35263
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:45.865671 3648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:45.865938 3648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:45.865967 3648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:45.867296 3648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:45.867347 3648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:45.867376 3648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:53:45.868924 3648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:45.869131 3648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:45.869330 3648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:45.875566 3654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:45.875574 3653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:45.875574 3656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:45.875612 3648 server_base.cc:1034] running on GCE node
I20260928 04:53:45.876181 3648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:45.876477 3648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:45.877647 3648 hybrid_clock.cc:648] HybridClock initialized: now 1790571225877625 us; error 42 us; skew 500 ppm
Sep 28 04:53:45 dist-test-slave-m39l krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:45.880399 3648 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:53:45.880960 3648 webserver.cc:460] Webserver started at http://127.0.98.2:40567/ using document root <none> and password file <none>
I20260928 04:53:45.881223 3648 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:45.881294 3648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:45.881450 3648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:45.882659 3648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "db80503330cd4d1eb9ec85a1e2a7a58b"
format_stamp: "Formatted at 2026-09-28 04:53:45 on dist-test-slave-m39l"
server_key: "3db408e3759f27233c40233b361adf3e"
server_key_iv: "fc85ab7e2799f3c3df6d468b423c2ce5"
server_key_version: "encryptionkey@0"
I20260928 04:53:45.883047 3648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "db80503330cd4d1eb9ec85a1e2a7a58b"
format_stamp: "Formatted at 2026-09-28 04:53:45 on dist-test-slave-m39l"
server_key: "3db408e3759f27233c40233b361adf3e"
server_key_iv: "fc85ab7e2799f3c3df6d468b423c2ce5"
server_key_version: "encryptionkey@0"
I20260928 04:53:45.884574 3648 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:53:45.885502 3663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:45.885802 3648 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:45.885948 3648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "db80503330cd4d1eb9ec85a1e2a7a58b"
format_stamp: "Formatted at 2026-09-28 04:53:45 on dist-test-slave-m39l"
server_key: "3db408e3759f27233c40233b361adf3e"
server_key_iv: "fc85ab7e2799f3c3df6d468b423c2ce5"
server_key_version: "encryptionkey@0"
I20260928 04:53:45.886037 3648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:45.895131 3648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:45.898008 3648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:45.898223 3648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:45.898581 3648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:45.898978 3648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:45.899019 3648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:45.899076 3648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:45.899114 3648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:45.906599 3648 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37565
I20260928 04:53:45.906656 3776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37565 every 8 connection(s)
I20260928 04:53:45.907053 3648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:53:45 dist-test-slave-m39l krb5kdc[3407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:45.917210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 3648
I20260928 04:53:45.917330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:53:45.917414 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:45.907879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38299 (local address 127.0.98.62:35263)
0928 04:53:45.907926 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:45.907928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:45.908208 (+ 280us) server_negotiation.cc:407] Connection header received
0928 04:53:45.908767 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:45.908768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:45.908782 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:45.908836 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:45.909872 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:45.910382 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:45.911514 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:45.911725 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:45.911816 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:45.914091 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:45.914094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:45.914094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:45.914101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:45.915808 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:45.916293 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:45.916294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:45.916295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:45.916356 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:45.916619 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:45.916620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:45.916621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:45.916758 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:45.916838 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:45.917199 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 04:53:45.917304 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:53:45.917544 392 external_mini_cluster.cc:1428] Setting key 179e22c95fb50d09166a09111c30f514
I20260928 04:53:45.917670 3779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:45.907884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35263 (local address 127.0.98.2:38299)
0928 04:53:45.908069 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:45.908113 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:45.908656 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:45.908927 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:45.908931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:45.909547 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:45.909759 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:45.909764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:45.910545 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:45.910546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:45.911413 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:45.911415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:45.911827 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:45.911829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:45.911963 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:45.912538 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:45.912547 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:45.913973 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:45.915899 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:45.915909 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:45.915911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:45.916197 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:45.916426 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:45.916427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:45.916427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:45.916543 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:45.916834 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:45.916842 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:45.917026 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 04:53:45.917156 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 04:53:45.917413 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260928 04:53:45.918130 3777 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35263
I20260928 04:53:45.918265 3777 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:45.918540 3777 heartbeater.cc:511] Master 127.0.98.62:35263 requested a full tablet report, sending...
I20260928 04:53:45.918975 3452 ts_manager.cc:194] Registered new tserver with Master: db80503330cd4d1eb9ec85a1e2a7a58b (127.0.98.2:37565)
I20260928 04:53:45.919360 3452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38299
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:53:45.946105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35263
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:46.035259 3782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:46.035537 3782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:46.035581 3782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:46.037003 3782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:46.037060 3782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:46.037091 3782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:53:46.038849 3782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:46.039134 3782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:46.039391 3782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:46.044754 3787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:46.044754 3788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:46.045200 3790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:46.045347 3782 server_base.cc:1034] running on GCE node
I20260928 04:53:46.045593 3782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:46.045827 3782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:46.047012 3782 hybrid_clock.cc:648] HybridClock initialized: now 1790571226046993 us; error 26 us; skew 500 ppm
Sep 28 04:53:46 dist-test-slave-m39l krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:46.049759 3782 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:53:46.050230 3782 webserver.cc:460] Webserver started at http://127.0.98.3:40361/ using document root <none> and password file <none>
I20260928 04:53:46.050460 3782 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:46.050578 3782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:46.050693 3782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:46.051956 3782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "d5431ec5b55c47038ac305fae04927f2"
format_stamp: "Formatted at 2026-09-28 04:53:46 on dist-test-slave-m39l"
server_key: "0360506055fe1c0c55ae5e74cd94252e"
server_key_iv: "1c4c0fe81626ddcf6534b4de61375d3f"
server_key_version: "encryptionkey@0"
I20260928 04:53:46.052307 3782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d5431ec5b55c47038ac305fae04927f2"
format_stamp: "Formatted at 2026-09-28 04:53:46 on dist-test-slave-m39l"
server_key: "0360506055fe1c0c55ae5e74cd94252e"
server_key_iv: "1c4c0fe81626ddcf6534b4de61375d3f"
server_key_version: "encryptionkey@0"
I20260928 04:53:46.053865 3782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:53:46.054863 3797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:46.055086 3782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:53:46.055184 3782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "d5431ec5b55c47038ac305fae04927f2"
format_stamp: "Formatted at 2026-09-28 04:53:46 on dist-test-slave-m39l"
server_key: "0360506055fe1c0c55ae5e74cd94252e"
server_key_iv: "1c4c0fe81626ddcf6534b4de61375d3f"
server_key_version: "encryptionkey@0"
I20260928 04:53:46.055262 3782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:46.080369 3782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:46.082592 3782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:46.082713 3782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:46.082906 3782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:46.083266 3782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:46.083302 3782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:46.083324 3782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:46.083338 3782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:46.089547 3782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44989
I20260928 04:53:46.089624 3910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44989 every 8 connection(s)
I20260928 04:53:46.089959 3782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:53:46.091490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 3782
I20260928 04:53:46.091578 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:53:46.091825 392 external_mini_cluster.cc:1428] Setting key 294a7a4a7fd436267f84745ee7be0f04
Sep 28 04:53:46 dist-test-slave-m39l krb5kdc[3407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:46.100716 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:46.090815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49801 (local address 127.0.98.62:35263)
0928 04:53:46.090888 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:46.090890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:46.091164 (+ 274us) server_negotiation.cc:407] Connection header received
0928 04:53:46.091745 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:46.091746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:46.091761 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:46.091826 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:46.093120 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:46.093632 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:46.094767 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:46.094986 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:46.095069 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:46.097612 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:46.097614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:46.097615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:46.097623 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:46.099146 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:46.099657 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:46.099659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:46.099660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:46.099725 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:46.100008 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:46.100009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:46.100010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:46.100140 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:46.100220 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:46.100526 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 04:53:46.100619 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:53:46.100920 3913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:46.090801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35263 (local address 127.0.98.3:49801)
0928 04:53:46.091012 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:46.091052 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:46.091619 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:46.091905 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:46.091910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:46.092743 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:46.093019 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:46.093025 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:46.093746 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:46.093746 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:46.094660 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:46.094662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:46.095078 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:46.095079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:46.095221 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:46.095826 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:46.095836 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:46.097499 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:46.099271 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:46.099287 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:46.099290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:46.099569 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:46.099815 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:46.099816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:46.099817 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:46.099939 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:46.100210 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:46.100217 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:46.100385 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 04:53:46.100493 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 04:53:46.100670 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
I20260928 04:53:46.101476 3911 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35263
I20260928 04:53:46.101569 3911 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:46.101783 3911 heartbeater.cc:511] Master 127.0.98.62:35263 requested a full tablet report, sending...
I20260928 04:53:46.102274 3452 ts_manager.cc:194] Registered new tserver with Master: d5431ec5b55c47038ac305fae04927f2 (127.0.98.3:44989)
I20260928 04:53:46.102801 3452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49801
I20260928 04:53:46.103797 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 04:53:46 dist-test-slave-m39l krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571226, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 04:53:46 dist-test-slave-m39l krb5kdc[3407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571226, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:46.128289 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:46.120264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39162 (local address 127.0.98.62:35263)
0928 04:53:46.120333 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:46.120335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:46.120414 (+ 79us) server_negotiation.cc:407] Connection header received
0928 04:53:46.120486 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:46.120486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:46.120497 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:46.120543 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:46.121242 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:46.121781 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:46.122641 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:46.122885 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:46.122954 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:46.125236 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:46.125238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:46.125239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:46.125246 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:46.126933 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:46.127295 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:46.127296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:46.127296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:46.127356 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:46.127619 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:46.127621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:46.127621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:46.127887 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:46.127984 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:46.128125 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 04:53:46.128215 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 04:53:46.128625 3452 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39162
I20260928 04:53:46.129263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 3514
I20260928 04:53:46.133668 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 3648
I20260928 04:53:46.138062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 3782
I20260928 04:53:46.142549 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 3420
2026-09-28T04:53:46Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2010 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:53:46 dist-test-slave-m39l krb5kdc[3923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:53:46 dist-test-slave-m39l krb5kdc[3923](info): set up 2 sockets
Sep 28 04:53:46 dist-test-slave-m39l krb5kdc[3923](info): commencing operation
krb5kdc: starting...
W20260928 04:53:47.190629 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:53:47 dist-test-slave-m39l krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:53:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:53:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:53:47.326571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41687
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:47.413555 3936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:47.413869 3936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:47.413919 3936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:47.415413 3936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:53:47.415508 3936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:47.415534 3936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:47.415561 3936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:53:47.415593 3936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:53:47.417353 3936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:47.417698 3936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:47.417970 3936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:47.423674 3941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:47.423708 3942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:47.423952 3936 server_base.cc:1034] running on GCE node
W20260928 04:53:47.423683 3944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:47.424340 3936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:47.424695 3936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:47.425863 3936 hybrid_clock.cc:648] HybridClock initialized: now 1790571227425847 us; error 58 us; skew 500 ppm
Sep 28 04:53:47 dist-test-slave-m39l krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:47.432999 3936 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:53:47.433517 3936 webserver.cc:460] Webserver started at http://127.0.98.62:35383/ using document root <none> and password file <none>
I20260928 04:53:47.433790 3936 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:47.433868 3936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:47.434023 3936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:47.435269 3936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "81e6812e6fff49a7bb62f136eee7e48c"
format_stamp: "Formatted at 2026-09-28 04:53:47 on dist-test-slave-m39l"
server_key: "0e3ac05aee03f0402fbc5fbee70a33bf"
server_key_iv: "0be34a5f4ae0dc5763902908f03f2737"
server_key_version: "encryptionkey@0"
I20260928 04:53:47.435658 3936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "81e6812e6fff49a7bb62f136eee7e48c"
format_stamp: "Formatted at 2026-09-28 04:53:47 on dist-test-slave-m39l"
server_key: "0e3ac05aee03f0402fbc5fbee70a33bf"
server_key_iv: "0be34a5f4ae0dc5763902908f03f2737"
server_key_version: "encryptionkey@0"
I20260928 04:53:47.437089 3936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:53:47.437891 3951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:47.438102 3936 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:47.438235 3936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "81e6812e6fff49a7bb62f136eee7e48c"
format_stamp: "Formatted at 2026-09-28 04:53:47 on dist-test-slave-m39l"
server_key: "0e3ac05aee03f0402fbc5fbee70a33bf"
server_key_iv: "0be34a5f4ae0dc5763902908f03f2737"
server_key_version: "encryptionkey@0"
I20260928 04:53:47.438316 3936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:47.446312 3936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:47.448737 3936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:47.448925 3936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:47.453022 3936 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41687
I20260928 04:53:47.453051 4003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41687 every 8 connection(s)
I20260928 04:53:47.453449 3936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:53:47.454319 4004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:47.456722 4004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c: Bootstrap starting.
I20260928 04:53:47.457263 4004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:47.457504 4004 log.cc:824] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:47.458151 4004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c: No bootstrap required, opened a new log
I20260928 04:53:47.459868 4004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e6812e6fff49a7bb62f136eee7e48c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41687 } }
I20260928 04:53:47.460021 4004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:47.460047 4004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e6812e6fff49a7bb62f136eee7e48c, State: Initialized, Role: FOLLOWER
I20260928 04:53:47.460197 4004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e6812e6fff49a7bb62f136eee7e48c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41687 } }
I20260928 04:53:47.460278 4004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:53:47.460299 4004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:53:47.460330 4004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:47.460836 4004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e6812e6fff49a7bb62f136eee7e48c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41687 } }
I20260928 04:53:47.460919 4004 leader_election.cc:302] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81e6812e6fff49a7bb62f136eee7e48c; no voters:
I20260928 04:53:47.461064 4004 leader_election.cc:288] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:53:47.461205 4007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:47.461374 4004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:47.461709 4007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c [term 1 LEADER]: Becoming Leader. State: Replica: 81e6812e6fff49a7bb62f136eee7e48c, State: Running, Role: LEADER
I20260928 04:53:47.461906 4007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e6812e6fff49a7bb62f136eee7e48c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41687 } }
I20260928 04:53:47.462333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 3936
I20260928 04:53:47.462469 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:53:47.462538 4009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e6812e6fff49a7bb62f136eee7e48c. Latest consensus state: current_term: 1 leader_uuid: "81e6812e6fff49a7bb62f136eee7e48c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e6812e6fff49a7bb62f136eee7e48c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41687 } } }
I20260928 04:53:47.462623 4009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:47.462678 392 external_mini_cluster.cc:1428] Setting key 2410ea70c429da6a05967594cd201995
I20260928 04:53:47.462651 4008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81e6812e6fff49a7bb62f136eee7e48c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e6812e6fff49a7bb62f136eee7e48c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41687 } } }
I20260928 04:53:47.462738 4008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c [sys.catalog]: This master's current role is: LEADER
W20260928 04:53:47.463481 4022 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:53:47.463573 4022 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:53:47.463681 4020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:53:47.463858 4020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:53:47.465395 4020 catalog_manager.cc:1347] Generated new cluster ID: cba0c0bc4d234b8c80abd43ea67733b1
I20260928 04:53:47.465468 4020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:53:47.478835 4020 catalog_manager.cc:1370] Generated new certificate authority record
Sep 28 04:53:47 dist-test-slave-m39l krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:47.479597 4020 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:53:47.484485 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:47.463292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46740 (local address 127.0.98.62:41687)
0928 04:53:47.463458 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:47.463469 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:47.463506 (+ 37us) server_negotiation.cc:407] Connection header received
0928 04:53:47.463646 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:47.463654 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:47.463717 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:47.463868 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:47.464636 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:47.475053 (+ 10417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:47.476266 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:47.476962 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:47.477071 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:47.480099 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:47.480106 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:47.480106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:47.480132 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:47.482281 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:47.482763 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:47.482767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:47.482767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:47.482889 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:47.483096 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:47.483097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:47.483097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:47.483411 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:47.483987 (+ 576us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:47.484063 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 04:53:47.484228 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260928 04:53:47.485139 4020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81e6812e6fff49a7bb62f136eee7e48c: Generated new TSK 0
I20260928 04:53:47.485307 4020 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:53:47.518193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41687
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:47.606591 4030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:47.606897 4030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:47.606945 4030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:47.608294 4030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:47.608389 4030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:47.608440 4030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:53:47.610270 4030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:47.610641 4030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:47.610948 4030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:47.616314 4035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:47.616487 4030 server_base.cc:1034] running on GCE node
W20260928 04:53:47.616333 4036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:47.616314 4038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:47.616874 4030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:47.617141 4030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:47.618335 4030 hybrid_clock.cc:648] HybridClock initialized: now 1790571227618307 us; error 37 us; skew 500 ppm
Sep 28 04:53:47 dist-test-slave-m39l krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:47.621148 4030 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:53:47.621659 4030 webserver.cc:460] Webserver started at http://127.0.98.1:34061/ using document root <none> and password file <none>
I20260928 04:53:47.621857 4030 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:47.621899 4030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:47.622000 4030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:47.623134 4030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "988f6db0a5cd41deb97a73035f14416f"
format_stamp: "Formatted at 2026-09-28 04:53:47 on dist-test-slave-m39l"
server_key: "8268b135b51bf4673e8d457c82081ebd"
server_key_iv: "83e7c0f4adb9e7b87e51eaadd974a30d"
server_key_version: "encryptionkey@0"
I20260928 04:53:47.623445 4030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "988f6db0a5cd41deb97a73035f14416f"
format_stamp: "Formatted at 2026-09-28 04:53:47 on dist-test-slave-m39l"
server_key: "8268b135b51bf4673e8d457c82081ebd"
server_key_iv: "83e7c0f4adb9e7b87e51eaadd974a30d"
server_key_version: "encryptionkey@0"
I20260928 04:53:47.624902 4030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:53:47.625829 4045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:47.626078 4030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:47.626199 4030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "988f6db0a5cd41deb97a73035f14416f"
format_stamp: "Formatted at 2026-09-28 04:53:47 on dist-test-slave-m39l"
server_key: "8268b135b51bf4673e8d457c82081ebd"
server_key_iv: "83e7c0f4adb9e7b87e51eaadd974a30d"
server_key_version: "encryptionkey@0"
I20260928 04:53:47.626283 4030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:47.647104 4030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:47.649655 4030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:47.649839 4030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:47.650102 4030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:47.650430 4030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:47.650485 4030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:47.650545 4030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:47.650599 4030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:47.657240 4030 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44901
I20260928 04:53:47.657285 4158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44901 every 8 connection(s)
I20260928 04:53:47.657630 4030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:53:47.664268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 4030
I20260928 04:53:47.664409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:53:47.664616 392 external_mini_cluster.cc:1428] Setting key a8429b1f9f31de4d14a76f56a8223497
Sep 28 04:53:47 dist-test-slave-m39l krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:47.670215 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:47.658466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57475 (local address 127.0.98.62:41687)
0928 04:53:47.658580 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:47.658582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:47.658732 (+ 150us) server_negotiation.cc:407] Connection header received
0928 04:53:47.659743 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:47.659744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:47.659756 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:47.659825 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:47.660521 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:47.661149 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:47.662450 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:47.662811 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:47.662877 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:47.667064 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:47.667066 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:47.667067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:47.667079 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:47.668606 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:47.669148 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:47.669149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:47.669150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:47.669239 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:47.669511 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:47.669512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:47.669513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:47.669637 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:47.669715 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:47.670009 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 04:53:47.670110 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 04:53:47.670432 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:47.658368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41687 (local address 127.0.98.1:57475)
0928 04:53:47.658596 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:47.658641 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:47.659556 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:47.659932 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:47.659936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:47.660206 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:47.660401 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:47.660406 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:47.661265 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:47.661266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:47.662340 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:47.662344 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:47.662967 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:47.662969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:47.663109 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:47.663822 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:47.663831 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:47.666947 (+ 3116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:47.668764 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:47.668774 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:47.668776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:47.669036 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:47.669314 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:47.669315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:47.669316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:47.669441 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:47.669706 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:47.669713 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:47.669871 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 04:53:47.669987 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 04:53:47.670177 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":100,"threads_started":1}
I20260928 04:53:47.671005 4159 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41687
I20260928 04:53:47.671106 4159 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:47.671348 4159 heartbeater.cc:511] Master 127.0.98.62:41687 requested a full tablet report, sending...
I20260928 04:53:47.671999 3968 ts_manager.cc:194] Registered new tserver with Master: 988f6db0a5cd41deb97a73035f14416f (127.0.98.1:44901)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 04:53:47.673015 3968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:53:47.695221 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41687
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:47.785863 4164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:47.786110 4164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:47.786139 4164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:47.787554 4164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:47.787616 4164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:47.787647 4164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:53:47.789290 4164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:47.789582 4164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:47.789844 4164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:47.795794 4170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:47.795866 4172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:47.795781 4169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:47.795926 4164 server_base.cc:1034] running on GCE node
I20260928 04:53:47.796375 4164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:47.796679 4164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:47.797881 4164 hybrid_clock.cc:648] HybridClock initialized: now 1790571227797858 us; error 49 us; skew 500 ppm
Sep 28 04:53:47 dist-test-slave-m39l krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:47.800843 4164 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:53:47.801393 4164 webserver.cc:460] Webserver started at http://127.0.98.2:41775/ using document root <none> and password file <none>
I20260928 04:53:47.801633 4164 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:47.801705 4164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:47.801872 4164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:47.803107 4164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "5b5fd45c53ef4665858b2692fc3c057b"
format_stamp: "Formatted at 2026-09-28 04:53:47 on dist-test-slave-m39l"
server_key: "c5855f5383a44344f5d1d794afc758bf"
server_key_iv: "fc54d3f6d67dfa9e4e304fa3ae7301f2"
server_key_version: "encryptionkey@0"
I20260928 04:53:47.803488 4164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5b5fd45c53ef4665858b2692fc3c057b"
format_stamp: "Formatted at 2026-09-28 04:53:47 on dist-test-slave-m39l"
server_key: "c5855f5383a44344f5d1d794afc758bf"
server_key_iv: "fc54d3f6d67dfa9e4e304fa3ae7301f2"
server_key_version: "encryptionkey@0"
I20260928 04:53:47.804975 4164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:53:47.805908 4179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:47.806119 4164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:47.806231 4164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "5b5fd45c53ef4665858b2692fc3c057b"
format_stamp: "Formatted at 2026-09-28 04:53:47 on dist-test-slave-m39l"
server_key: "c5855f5383a44344f5d1d794afc758bf"
server_key_iv: "fc54d3f6d67dfa9e4e304fa3ae7301f2"
server_key_version: "encryptionkey@0"
I20260928 04:53:47.806337 4164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:47.819096 4164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:47.821635 4164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:47.821822 4164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:47.822090 4164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:47.822414 4164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:47.822469 4164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:47.822562 4164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:47.822611 4164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:47.828800 4164 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36953
I20260928 04:53:47.828913 4292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36953 every 8 connection(s)
I20260928 04:53:47.829279 4164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:53:47.830883 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 4164
I20260928 04:53:47.831001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:53:47.831213 392 external_mini_cluster.cc:1428] Setting key efaf7579a98e696edffbfdbe85ed7295
Sep 28 04:53:47 dist-test-slave-m39l krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 04:53:47.840811 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:47.830092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53629 (local address 127.0.98.62:41687)
0928 04:53:47.830140 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:47.830142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:47.830342 (+ 200us) server_negotiation.cc:407] Connection header received
0928 04:53:47.831684 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:47.831685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:47.831696 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:47.831740 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:47.832684 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:47.833202 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:47.834338 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:47.834683 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:47.834756 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:47.837214 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:47.837216 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:47.837217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:47.837226 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:47.839119 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:47.839567 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:47.839568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:47.839569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:47.839633 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:47.839950 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:47.839951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:47.839952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:47.840108 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:47.840181 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:47.840598 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 04:53:47.840704 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:53:47.841042 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:47.830060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41687 (local address 127.0.98.2:53629)
0928 04:53:47.830222 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:47.830266 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:47.831555 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:47.831820 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:47.831825 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:47.832239 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:47.832552 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:47.832559 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:47.833320 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:47.833322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:47.834249 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:47.834251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:47.834776 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:47.834777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:47.834905 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:47.835471 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:47.835480 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:47.837106 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:47.839225 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:47.839232 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:47.839234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:47.839481 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:47.839765 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:47.839766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:47.839766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:47.839873 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:47.840236 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:47.840242 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:47.840437 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 04:53:47.840559 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 04:53:47.840777 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260928 04:53:47.841647 4293 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41687
I20260928 04:53:47.841912 4293 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:47.842139 4293 heartbeater.cc:511] Master 127.0.98.62:41687 requested a full tablet report, sending...
I20260928 04:53:47.842635 3968 ts_manager.cc:194] Registered new tserver with Master: 5b5fd45c53ef4665858b2692fc3c057b (127.0.98.2:36953)
I20260928 04:53:47.843096 3968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:53:47.862907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41687
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:47.946903 4298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:47.947234 4298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:47.947297 4298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:47.948508 4298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:47.948578 4298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:47.948627 4298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:53:47.950284 4298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:47.950709 4298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:47.951020 4298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:47.956354 4303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:47.956354 4306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:47.956410 4298 server_base.cc:1034] running on GCE node
W20260928 04:53:47.956354 4304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:47.956822 4298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:47.957041 4298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:47.958168 4298 hybrid_clock.cc:648] HybridClock initialized: now 1790571227958150 us; error 31 us; skew 500 ppm
Sep 28 04:53:47 dist-test-slave-m39l krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:47.960878 4298 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:53:47.961385 4298 webserver.cc:460] Webserver started at http://127.0.98.3:44243/ using document root <none> and password file <none>
I20260928 04:53:47.961637 4298 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:47.961710 4298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:47.961871 4298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:47.963280 4298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "33f4dc4fb6ac4f1c9367b6fcb2169a73"
format_stamp: "Formatted at 2026-09-28 04:53:47 on dist-test-slave-m39l"
server_key: "222cbbb67d232ab1da942d4768d5d2b5"
server_key_iv: "3447d00e108e8706f5b00f9990e1271c"
server_key_version: "encryptionkey@0"
I20260928 04:53:47.963660 4298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "33f4dc4fb6ac4f1c9367b6fcb2169a73"
format_stamp: "Formatted at 2026-09-28 04:53:47 on dist-test-slave-m39l"
server_key: "222cbbb67d232ab1da942d4768d5d2b5"
server_key_iv: "3447d00e108e8706f5b00f9990e1271c"
server_key_version: "encryptionkey@0"
I20260928 04:53:47.965101 4298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:53:47.965992 4313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:47.966238 4298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:47.966351 4298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "33f4dc4fb6ac4f1c9367b6fcb2169a73"
format_stamp: "Formatted at 2026-09-28 04:53:47 on dist-test-slave-m39l"
server_key: "222cbbb67d232ab1da942d4768d5d2b5"
server_key_iv: "3447d00e108e8706f5b00f9990e1271c"
server_key_version: "encryptionkey@0"
I20260928 04:53:47.966430 4298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:47.975976 4298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:47.978565 4298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:47.978739 4298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:47.978992 4298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:47.979326 4298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:47.979382 4298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:47.979427 4298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:47.979453 4298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:47.986097 4298 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42255
I20260928 04:53:47.986203 4426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42255 every 8 connection(s)
I20260928 04:53:47.986619 4298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:53:47.988780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 4298
I20260928 04:53:47.988860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:53:47.989171 392 external_mini_cluster.cc:1428] Setting key 0806919c5709009bf0be076d42fff89f
Sep 28 04:53:47 dist-test-slave-m39l krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:47.998219 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:47.987546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50849 (local address 127.0.98.62:41687)
0928 04:53:47.987632 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:47.987635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:47.987820 (+ 185us) server_negotiation.cc:407] Connection header received
0928 04:53:47.989093 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:47.989095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:47.989112 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:47.989193 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:47.990065 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:47.990782 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:47.992378 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:47.992595 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:47.992731 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:47.995160 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:47.995162 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:47.995162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:47.995171 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:47.996757 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:47.997141 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:47.997142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:47.997142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:47.997221 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:47.997441 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:47.997441 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:47.997441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:47.997568 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:47.997625 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:47.998049 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 04:53:47.998128 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 04:53:47.998466 4429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:47.987501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41687 (local address 127.0.98.3:50849)
0928 04:53:47.987687 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:47.987726 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:47.988637 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:47.989332 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:47.989337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:47.989731 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:47.989966 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:47.989972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:47.991004 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:47.991005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:47.992268 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:47.992271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:47.992775 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:47.992777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:47.992899 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:47.993428 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:47.993437 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:47.995057 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:47.996839 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:47.996845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:47.996847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:47.997071 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:47.997283 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:47.997284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:47.997284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:47.997384 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:47.997766 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:47.997773 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:47.997909 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 04:53:47.998013 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 04:53:47.998149 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
I20260928 04:53:47.999122 4427 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41687
I20260928 04:53:47.999253 4427 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:47.999445 4427 heartbeater.cc:511] Master 127.0.98.62:41687 requested a full tablet report, sending...
I20260928 04:53:48.000058 3968 ts_manager.cc:194] Registered new tserver with Master: 33f4dc4fb6ac4f1c9367b6fcb2169a73 (127.0.98.3:42255)
I20260928 04:53:48.000524 3968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50849
I20260928 04:53:48.002245 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 04:53:48 dist-test-slave-m39l krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 04:53:48 dist-test-slave-m39l krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:53:48.028151 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:48.019027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47924 (local address 127.0.98.1:44901)
0928 04:53:48.019265 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:48.019270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:48.019282 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:53:48.019319 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:48.019321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:48.019337 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:48.019413 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:48.020111 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:48.020956 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:48.021786 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:48.022149 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:48.022268 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:48.024944 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:48.024953 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:48.024953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:48.024960 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:48.026866 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:48.027277 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:48.027281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:48.027281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:48.027356 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:48.027565 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:48.027566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:48.027566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:48.027722 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:48.027893 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:48.027938 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 04:53:48.028023 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":104,"threads_started":1}
Sep 28 04:53:48 dist-test-slave-m39l krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:48.041620 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:48.032713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46754 (local address 127.0.98.62:41687)
0928 04:53:48.032763 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:48.032765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:48.033247 (+ 482us) server_negotiation.cc:407] Connection header received
0928 04:53:48.033333 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:48.033334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:48.033347 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:48.033402 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:48.033983 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:48.034889 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:48.035997 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:48.036301 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:48.036389 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:48.039004 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:48.039006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:48.039006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:48.039013 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:48.040543 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:48.040870 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:48.040871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:48.040871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:48.040934 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:48.041115 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:48.041116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:48.041116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:48.041306 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:48.041380 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:48.041477 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 04:53:48.041546 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 04:53:48.042019 3968 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46754
I20260928 04:53:48.042430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 4030
I20260928 04:53:48.048382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 4164
I20260928 04:53:48.052975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 4298
I20260928 04:53:48.057730 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 3936
2026-09-28T04:53:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1914 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:53:48 dist-test-slave-m39l krb5kdc[4439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:53:48 dist-test-slave-m39l krb5kdc[4439](info): set up 2 sockets
Sep 28 04:53:48 dist-test-slave-m39l krb5kdc[4439](info): commencing operation
krb5kdc: starting...
W20260928 04:53:49.138622 392 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:53:49 dist-test-slave-m39l krb5kdc[4439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:53:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:53:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:53:49.274085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46883
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:49.356113 4452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:49.356403 4452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:49.356446 4452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:49.357707 4452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:53:49.357788 4452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:49.357813 4452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:49.357836 4452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:53:49.357863 4452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:53:49.359472 4452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:49.359779 4452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:49.360023 4452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:49.365250 4457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:49.365288 4458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:49.365284 4460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:49.365571 4452 server_base.cc:1034] running on GCE node
I20260928 04:53:49.365840 4452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:49.366117 4452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:49.367286 4452 hybrid_clock.cc:648] HybridClock initialized: now 1790571229367269 us; error 28 us; skew 500 ppm
Sep 28 04:53:49 dist-test-slave-m39l krb5kdc[4439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:49.370059 4452 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:53:49.370426 4452 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:49.370528 4452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:49.370685 4452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:49.371809 4452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "ee32bac3b88645d7b28f2998e7d7eae7"
format_stamp: "Formatted at 2026-09-28 04:53:49 on dist-test-slave-m39l"
server_key: "6cf8139a009acddf783ec59f1c6b9e2e"
server_key_iv: "fa49a46be3a191395a7e6d5b2c68ab3a"
server_key_version: "encryptionkey@0"
I20260928 04:53:49.372174 4452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee32bac3b88645d7b28f2998e7d7eae7"
format_stamp: "Formatted at 2026-09-28 04:53:49 on dist-test-slave-m39l"
server_key: "6cf8139a009acddf783ec59f1c6b9e2e"
server_key_iv: "fa49a46be3a191395a7e6d5b2c68ab3a"
server_key_version: "encryptionkey@0"
I20260928 04:53:49.373519 4452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:53:49.374315 4466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:49.374485 4452 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:53:49.374634 4452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "ee32bac3b88645d7b28f2998e7d7eae7"
format_stamp: "Formatted at 2026-09-28 04:53:49 on dist-test-slave-m39l"
server_key: "6cf8139a009acddf783ec59f1c6b9e2e"
server_key_iv: "fa49a46be3a191395a7e6d5b2c68ab3a"
server_key_version: "encryptionkey@0"
I20260928 04:53:49.374775 4452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:49.391386 4452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:49.393659 4452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:49.393848 4452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:49.398785 4452 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46883
I20260928 04:53:49.398833 4518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46883 every 8 connection(s)
I20260928 04:53:49.399186 4452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:53:49.400192 4519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:49.402714 4519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7: Bootstrap starting.
I20260928 04:53:49.403410 4519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:49.403741 4519 log.cc:824] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:49.404572 4519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7: No bootstrap required, opened a new log
I20260928 04:53:49.406256 4519 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee32bac3b88645d7b28f2998e7d7eae7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46883 } }
I20260928 04:53:49.406423 4519 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:49.406447 4519 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee32bac3b88645d7b28f2998e7d7eae7, State: Initialized, Role: FOLLOWER
I20260928 04:53:49.406652 4519 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee32bac3b88645d7b28f2998e7d7eae7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46883 } }
I20260928 04:53:49.406734 4519 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:53:49.406759 4519 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:53:49.406828 4519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:49.407739 4519 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee32bac3b88645d7b28f2998e7d7eae7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46883 } }
I20260928 04:53:49.407861 4519 leader_election.cc:302] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee32bac3b88645d7b28f2998e7d7eae7; no voters:
I20260928 04:53:49.408077 4519 leader_election.cc:288] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:53:49.408200 4522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:49.408373 4522 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7 [term 1 LEADER]: Becoming Leader. State: Replica: ee32bac3b88645d7b28f2998e7d7eae7, State: Running, Role: LEADER
I20260928 04:53:49.408396 4519 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:49.408478 4522 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee32bac3b88645d7b28f2998e7d7eae7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46883 } }
I20260928 04:53:49.408893 4524 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee32bac3b88645d7b28f2998e7d7eae7. Latest consensus state: current_term: 1 leader_uuid: "ee32bac3b88645d7b28f2998e7d7eae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee32bac3b88645d7b28f2998e7d7eae7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46883 } } }
I20260928 04:53:49.408890 4523 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee32bac3b88645d7b28f2998e7d7eae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee32bac3b88645d7b28f2998e7d7eae7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46883 } } }
I20260928 04:53:49.408999 4524 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:49.409005 4523 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:49.409233 4531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:53:49.409361 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 4452
I20260928 04:53:49.409483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:53:49.409727 392 external_mini_cluster.cc:1428] Setting key 46d239b02ab0e7f55214efb53641b404
I20260928 04:53:49.409869 4531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:53:49.411892 4531 catalog_manager.cc:1347] Generated new cluster ID: b5dc7eb1eb0a41aa821e5302cf022dc0
I20260928 04:53:49.411968 4531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:53:49.416826 4531 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:53:49.417549 4531 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 04:53:49 dist-test-slave-m39l krb5kdc[4439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:49.422680 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:49.410663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34430 (local address 127.0.98.62:46883)
0928 04:53:49.410834 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:49.410844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:49.410885 (+ 41us) server_negotiation.cc:407] Connection header received
0928 04:53:49.411009 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:49.411017 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:49.411096 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:49.411243 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:49.412240 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:49.413735 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:49.414809 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:49.415500 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:49.415621 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:49.418695 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:49.418700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:49.418701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:49.418714 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:49.420834 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:49.421329 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:49.421333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:49.421333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:49.421455 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:49.421680 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:49.421681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:49.421681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:49.421968 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:49.422114 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:49.422241 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 04:53:49.422390 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260928 04:53:49.428079 4531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7: Generated new TSK 0
I20260928 04:53:49.428288 4531 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:53:49.466259 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46883
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:49.565989 4545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:49.566238 4545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:49.566267 4545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:49.567696 4545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:49.567760 4545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:49.567793 4545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:53:49.569491 4545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:49.569710 4545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:49.569914 4545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:49.575469 4551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:49.575511 4550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:49.575476 4553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:49.575950 4545 server_base.cc:1034] running on GCE node
I20260928 04:53:49.576215 4545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:49.576494 4545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:49.577760 4545 hybrid_clock.cc:648] HybridClock initialized: now 1790571229577749 us; error 39 us; skew 500 ppm
Sep 28 04:53:49 dist-test-slave-m39l krb5kdc[4439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:49.580991 4545 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:53:49.581389 4545 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:49.581467 4545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:49.581633 4545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:49.582844 4545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "9585c0a980e245f9833172cccbd446df"
format_stamp: "Formatted at 2026-09-28 04:53:49 on dist-test-slave-m39l"
server_key: "0020c9024fa67b5450fa37bfa1703c64"
server_key_iv: "7498ca7431ef93a4f71a6560cdd787c5"
server_key_version: "encryptionkey@0"
I20260928 04:53:49.583215 4545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9585c0a980e245f9833172cccbd446df"
format_stamp: "Formatted at 2026-09-28 04:53:49 on dist-test-slave-m39l"
server_key: "0020c9024fa67b5450fa37bfa1703c64"
server_key_iv: "7498ca7431ef93a4f71a6560cdd787c5"
server_key_version: "encryptionkey@0"
I20260928 04:53:49.584698 4545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:53:49.585569 4559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:49.585764 4545 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:49.585875 4545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "9585c0a980e245f9833172cccbd446df"
format_stamp: "Formatted at 2026-09-28 04:53:49 on dist-test-slave-m39l"
server_key: "0020c9024fa67b5450fa37bfa1703c64"
server_key_iv: "7498ca7431ef93a4f71a6560cdd787c5"
server_key_version: "encryptionkey@0"
I20260928 04:53:49.585953 4545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:49.606377 4545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:49.608834 4545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:49.609066 4545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:49.609351 4545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:49.609724 4545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:49.609782 4545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:49.609822 4545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:49.609851 4545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:49.616246 4545 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45247
I20260928 04:53:49.616323 4672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45247 every 8 connection(s)
I20260928 04:53:49.616644 4545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:53:49.622748 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 4545
I20260928 04:53:49.622850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:53:49.623055 392 external_mini_cluster.cc:1428] Setting key 2a0ae328658c517e7ad01d958b5a164e
Sep 28 04:53:49 dist-test-slave-m39l krb5kdc[4439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:49.627446 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:49.617508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40773 (local address 127.0.98.62:46883)
0928 04:53:49.617706 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:49.617710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:49.617821 (+ 111us) server_negotiation.cc:407] Connection header received
0928 04:53:49.618336 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:49.618337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:49.618357 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:49.618434 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:49.619435 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:49.620029 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:49.621114 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:49.621326 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:49.621396 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:49.624148 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:49.624150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:49.624150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:49.624159 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:49.625789 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:49.626266 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:49.626267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:49.626268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:49.626335 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:49.626725 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:49.626726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:49.626726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:49.626863 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:49.626947 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:49.627250 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 04:53:49.627349 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260928 04:53:49.627750 4675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:49.617479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46883 (local address 127.0.98.1:40773)
0928 04:53:49.617653 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:49.617705 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:49.618216 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:49.618513 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:49.618517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:49.618797 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:49.619314 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:49.619320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:49.620171 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:49.620172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:49.621005 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:49.621008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:49.621423 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:49.621424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:49.621538 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:49.622184 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:49.622194 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:49.624036 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:49.625908 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:49.625918 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:49.625921 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:49.626181 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:49.626406 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:49.626407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:49.626408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:49.626600 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:49.626935 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:49.626943 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:49.627099 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 04:53:49.627214 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 04:53:49.627404 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260928 04:53:49.628368 4673 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46883
I20260928 04:53:49.628484 4673 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:49.628662 4673 heartbeater.cc:511] Master 127.0.98.62:46883 requested a full tablet report, sending...
I20260928 04:53:49.629238 4483 ts_manager.cc:194] Registered new tserver with Master: 9585c0a980e245f9833172cccbd446df (127.0.98.1:45247)
I20260928 04:53:49.629899 4483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40773
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:53:49.655579 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46883
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:49.748381 4678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:49.748613 4678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:49.748639 4678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:49.749795 4678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:49.749835 4678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:49.749863 4678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:53:49.751538 4678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:49.751745 4678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:49.751940 4678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:49.757472 4686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:49.757457 4678 server_base.cc:1034] running on GCE node
W20260928 04:53:49.757457 4684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:49.757467 4683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:49.758033 4678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:49.758307 4678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:49.759472 4678 hybrid_clock.cc:648] HybridClock initialized: now 1790571229759448 us; error 40 us; skew 500 ppm
Sep 28 04:53:49 dist-test-slave-m39l krb5kdc[4439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:49.762418 4678 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:53:49.762812 4678 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:49.762892 4678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:49.763046 4678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:49.764128 4678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "a15a08863df049b6bb5141ed6770598b"
format_stamp: "Formatted at 2026-09-28 04:53:49 on dist-test-slave-m39l"
server_key: "f9068a0d8430c1aa1e4871ffaee994d3"
server_key_iv: "0f7f569334f1f6d4dbe3c3567cf88134"
server_key_version: "encryptionkey@0"
I20260928 04:53:49.764494 4678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a15a08863df049b6bb5141ed6770598b"
format_stamp: "Formatted at 2026-09-28 04:53:49 on dist-test-slave-m39l"
server_key: "f9068a0d8430c1aa1e4871ffaee994d3"
server_key_iv: "0f7f569334f1f6d4dbe3c3567cf88134"
server_key_version: "encryptionkey@0"
I20260928 04:53:49.765925 4678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:53:49.766878 4692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:49.767105 4678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:49.767227 4678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "a15a08863df049b6bb5141ed6770598b"
format_stamp: "Formatted at 2026-09-28 04:53:49 on dist-test-slave-m39l"
server_key: "f9068a0d8430c1aa1e4871ffaee994d3"
server_key_iv: "0f7f569334f1f6d4dbe3c3567cf88134"
server_key_version: "encryptionkey@0"
I20260928 04:53:49.767309 4678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:49.804111 4678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:49.806612 4678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:49.806844 4678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:49.807137 4678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:49.807530 4678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:49.807590 4678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:49.807629 4678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:49.807683 4678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:49.815403 4678 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33103
I20260928 04:53:49.815433 4805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33103 every 8 connection(s)
I20260928 04:53:49.815827 4678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:53:49.821630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 4678
I20260928 04:53:49.821815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:53:49.822080 392 external_mini_cluster.cc:1428] Setting key d32ca027ae1aeb8034625bd584c3bef9
Sep 28 04:53:49 dist-test-slave-m39l krb5kdc[4439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:49.827744 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:49.816756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47291 (local address 127.0.98.62:46883)
0928 04:53:49.816852 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:49.816854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:49.817073 (+ 219us) server_negotiation.cc:407] Connection header received
0928 04:53:49.817596 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:49.817598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:49.817611 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:49.817688 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:49.818727 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:49.819257 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:49.820399 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:49.820597 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:49.820722 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:49.824177 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:49.824178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:49.824179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:49.824187 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:49.826067 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:49.826572 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:49.826574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:49.826574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:49.826642 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:49.826931 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:49.826932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:49.826933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:49.827099 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:49.827181 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:49.827531 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 04:53:49.827629 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 04:53:49.827900 4808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:49.816726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46883 (local address 127.0.98.2:47291)
0928 04:53:49.816930 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:49.816976 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:49.817485 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:49.817745 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:49.817749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:49.818050 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:49.818602 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:49.818608 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:49.819379 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:49.819380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:49.820287 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:49.820290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:49.820776 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:49.820777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:49.820932 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:49.821779 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:49.821790 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:49.824044 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:49.826174 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:49.826182 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:49.826185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:49.826438 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:49.826713 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:49.826713 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:49.826714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:49.826846 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:49.827195 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:49.827203 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:49.827367 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 04:53:49.827462 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 04:53:49.827669 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":107,"threads_started":1}
I20260928 04:53:49.828476 4806 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46883
I20260928 04:53:49.828572 4806 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:49.828857 4806 heartbeater.cc:511] Master 127.0.98.62:46883 requested a full tablet report, sending...
I20260928 04:53:49.829353 4483 ts_manager.cc:194] Registered new tserver with Master: a15a08863df049b6bb5141ed6770598b (127.0.98.2:33103)
I20260928 04:53:49.829893 4483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47291
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:53:49.850454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46883
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:49.933665 4811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:49.933916 4811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:49.933960 4811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:49.935372 4811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:49.935446 4811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:49.935493 4811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:53:49.937263 4811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:49.937570 4811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:49.937825 4811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:49.943662 4816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:49.943723 4817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:49.943650 4819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:49.943975 4811 server_base.cc:1034] running on GCE node
I20260928 04:53:49.944293 4811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:49.944641 4811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:49.945842 4811 hybrid_clock.cc:648] HybridClock initialized: now 1790571229945835 us; error 42 us; skew 500 ppm
Sep 28 04:53:49 dist-test-slave-m39l krb5kdc[4439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:49.948518 4811 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:53:49.948899 4811 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:49.948974 4811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:49.949122 4811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:49.950197 4811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "051acf36e89c4389a8a781c6603fece0"
format_stamp: "Formatted at 2026-09-28 04:53:49 on dist-test-slave-m39l"
server_key: "044e8ab28a672e5dc0976921f1bbd223"
server_key_iv: "246834086e91e8362880eca8aaa8a1d5"
server_key_version: "encryptionkey@0"
I20260928 04:53:49.950619 4811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "051acf36e89c4389a8a781c6603fece0"
format_stamp: "Formatted at 2026-09-28 04:53:49 on dist-test-slave-m39l"
server_key: "044e8ab28a672e5dc0976921f1bbd223"
server_key_iv: "246834086e91e8362880eca8aaa8a1d5"
server_key_version: "encryptionkey@0"
I20260928 04:53:49.951949 4811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:49.952785 4825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:49.953006 4811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:49.953101 4811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "051acf36e89c4389a8a781c6603fece0"
format_stamp: "Formatted at 2026-09-28 04:53:49 on dist-test-slave-m39l"
server_key: "044e8ab28a672e5dc0976921f1bbd223"
server_key_iv: "246834086e91e8362880eca8aaa8a1d5"
server_key_version: "encryptionkey@0"
I20260928 04:53:49.953187 4811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:49.966876 4811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:49.969694 4811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:49.969985 4811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:49.970315 4811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:49.970763 4811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:49.970824 4811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:49.970871 4811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:49.970902 4811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:49.977838 4811 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43879
I20260928 04:53:49.977972 4938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43879 every 8 connection(s)
I20260928 04:53:49.978266 4811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:53:49 dist-test-slave-m39l krb5kdc[4439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:49.985929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 4811
I20260928 04:53:49.986052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:53:49.986274 392 external_mini_cluster.cc:1428] Setting key 2e64a098a04d0477eabd430bdb91f809
I20260928 04:53:49.989141 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:49.979217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34199 (local address 127.0.98.62:46883)
0928 04:53:49.979294 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:49.979297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:49.979551 (+ 254us) server_negotiation.cc:407] Connection header received
0928 04:53:49.980195 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:49.980196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:49.980207 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:49.980265 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:49.981350 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:49.981854 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:49.983097 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:49.983294 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:49.983349 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:49.985754 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:49.985755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:49.985755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:49.985761 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:49.987441 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:49.987946 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:49.987947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:49.987947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:49.988000 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:49.988256 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:49.988257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:49.988258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:49.988391 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:49.988545 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:49.988933 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 04:53:49.989029 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 04:53:49.989387 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:49.979191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46883 (local address 127.0.98.3:34199)
0928 04:53:49.979416 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:49.979456 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:49.980056 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:49.980356 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:49.980361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:49.980718 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:49.981266 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:49.981271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:49.981952 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:49.981953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:49.983018 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:49.983021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:49.983377 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:49.983378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:49.983491 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:49.984165 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:49.984174 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:49.985667 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:49.987513 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:49.987519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:49.987521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:49.987854 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:49.988048 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:49.988048 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:49.988049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:49.988159 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:49.988587 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:49.988594 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:49.988771 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 04:53:49.988908 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 04:53:49.989121 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":118,"threads_started":1}
I20260928 04:53:49.989966 4939 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46883
I20260928 04:53:49.990074 4939 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:49.990257 4939 heartbeater.cc:511] Master 127.0.98.62:46883 requested a full tablet report, sending...
I20260928 04:53:49.990763 4483 ts_manager.cc:194] Registered new tserver with Master: 051acf36e89c4389a8a781c6603fece0 (127.0.98.3:43879)
I20260928 04:53:49.991196 4483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34199
I20260928 04:53:49.998901 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:53:50.007296 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.000298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.0.98.62:46883)
0928 04:53:50.000371 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:50.000372 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:50.000535 (+ 163us) server_negotiation.cc:407] Connection header received
0928 04:53:50.000696 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:50.000697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:50.000708 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:50.000769 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:50.001375 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.001865 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.002765 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.002985 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.003083 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:50.004475 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:50.004476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:50.004477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:50.004485 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:50.006107 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:50.006571 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:50.006577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:50.006578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:50.006640 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:50.006827 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:50.006828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:50.006828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:50.006982 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:50.007056 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:50.007154 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 04:53:50.007226 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 04:53:50.009191 4483 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:53:50.010941 4483 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:53:50.020557 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.015125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57982 (local address 127.0.98.1:45247)
0928 04:53:50.015954 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:50.015959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:50.015970 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:53:50.016032 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:50.016035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:50.016051 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:50.016424 (+ 373us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:50.016500 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.017677 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.019301 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.020081 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.020200 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:50.020269 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:50.020425 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 04:53:50.020466 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":193,"threads_started":1}
I20260928 04:53:50.020568 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.014627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45247 (local address 127.0.0.1:57982)
0928 04:53:50.015753 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:50.015761 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:50.015794 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:50.016202 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:50.016205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:50.016208 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:50.016308 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:50.016311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.017797 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.017798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:50.019193 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:50.019196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.020183 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.020184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:50.020293 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:50.020311 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 04:53:50.020393 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 04:53:50.020460 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1090,"thread_start_us":98,"threads_started":1}
I20260928 04:53:50.021543 4607 tablet_service.cc:1467] Processing CreateTablet for tablet d0e554c42ba347d19d912a7a99da7b79 (DEFAULT_TABLE table=test-table [id=9296ae0879c9439782ba84e4bf9115a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:50.021917 4607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e554c42ba347d19d912a7a99da7b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:50.022522 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.014832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33103 (local address 127.0.0.1:58152)
0928 04:53:50.018290 (+ 3458us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:50.018302 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:50.018366 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:50.019237 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:50.019238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:50.019241 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:50.019292 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:50.019294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.020231 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.020232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:50.021357 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:50.021359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.022226 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.022227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:50.022300 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:50.022306 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 04:53:50.022363 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 04:53:50.022404 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3440,"thread_start_us":76,"threads_started":1}
I20260928 04:53:50.022567 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.018684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58152 (local address 127.0.98.2:33103)
0928 04:53:50.018995 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:50.019000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:50.019010 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:53:50.019054 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:50.019056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:50.019073 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:50.019150 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:50.019374 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.020082 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.021449 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.022130 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.022215 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:50.022270 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:50.022419 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 04:53:50.022459 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":77,"threads_started":1}
I20260928 04:53:50.023502 4740 tablet_service.cc:1467] Processing CreateTablet for tablet d0e554c42ba347d19d912a7a99da7b79 (DEFAULT_TABLE table=test-table [id=9296ae0879c9439782ba84e4bf9115a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:50.023826 4740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e554c42ba347d19d912a7a99da7b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:50.024263 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.014082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43879 (local address 127.0.0.1:55916)
0928 04:53:50.015441 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:50.015513 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:50.015608 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:50.020447 (+ 4839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:50.020448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:50.020450 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:50.020484 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:50.020486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.022012 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.022013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:50.023131 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:50.023133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.024014 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.024015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:50.024077 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:50.024081 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:53:50.024137 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 04:53:50.024183 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1324,"thread_start_us":83,"threads_started":1}
I20260928 04:53:50.024283 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.018615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.0.98.3:43879)
0928 04:53:50.020277 (+ 1662us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:50.020282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:50.020292 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:53:50.020329 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:50.020331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:50.020348 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:50.020565 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:50.020652 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.021903 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.023202 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.023923 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.023996 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:50.024050 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:50.024167 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 04:53:50.024206 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1619,"thread_start_us":99,"threads_started":1}
I20260928 04:53:50.025032 4873 tablet_service.cc:1467] Processing CreateTablet for tablet d0e554c42ba347d19d912a7a99da7b79 (DEFAULT_TABLE table=test-table [id=9296ae0879c9439782ba84e4bf9115a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:50.025350 4873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e554c42ba347d19d912a7a99da7b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:50.026015 4956 tablet_bootstrap.cc:492] T d0e554c42ba347d19d912a7a99da7b79 P 9585c0a980e245f9833172cccbd446df: Bootstrap starting.
I20260928 04:53:50.026736 4956 tablet_bootstrap.cc:654] T d0e554c42ba347d19d912a7a99da7b79 P 9585c0a980e245f9833172cccbd446df: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:50.027391 4957 tablet_bootstrap.cc:492] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b: Bootstrap starting.
I20260928 04:53:50.027450 4956 log.cc:824] T d0e554c42ba347d19d912a7a99da7b79 P 9585c0a980e245f9833172cccbd446df: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:50.028064 4957 tablet_bootstrap.cc:654] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:50.028265 4956 tablet_bootstrap.cc:492] T d0e554c42ba347d19d912a7a99da7b79 P 9585c0a980e245f9833172cccbd446df: No bootstrap required, opened a new log
I20260928 04:53:50.028344 4956 ts_tablet_manager.cc:1397] T d0e554c42ba347d19d912a7a99da7b79 P 9585c0a980e245f9833172cccbd446df: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:53:50.028730 4957 log.cc:824] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:50.028987 4959 tablet_bootstrap.cc:492] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0: Bootstrap starting.
I20260928 04:53:50.029541 4957 tablet_bootstrap.cc:492] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b: No bootstrap required, opened a new log
I20260928 04:53:50.029609 4957 ts_tablet_manager.cc:1397] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:53:50.030834 4959 tablet_bootstrap.cc:654] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:50.030723 4956 raft_consensus.cc:348] T d0e554c42ba347d19d912a7a99da7b79 P 9585c0a980e245f9833172cccbd446df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051acf36e89c4389a8a781c6603fece0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43879 } } peers { permanent_uuid: "9585c0a980e245f9833172cccbd446df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45247 } } peers { permanent_uuid: "a15a08863df049b6bb5141ed6770598b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33103 } }
I20260928 04:53:50.030892 4956 raft_consensus.cc:374] T d0e554c42ba347d19d912a7a99da7b79 P 9585c0a980e245f9833172cccbd446df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:50.030930 4956 raft_consensus.cc:729] T d0e554c42ba347d19d912a7a99da7b79 P 9585c0a980e245f9833172cccbd446df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9585c0a980e245f9833172cccbd446df, State: Initialized, Role: FOLLOWER
I20260928 04:53:50.031086 4956 consensus_queue.cc:260] T d0e554c42ba347d19d912a7a99da7b79 P 9585c0a980e245f9833172cccbd446df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051acf36e89c4389a8a781c6603fece0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43879 } } peers { permanent_uuid: "9585c0a980e245f9833172cccbd446df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45247 } } peers { permanent_uuid: "a15a08863df049b6bb5141ed6770598b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33103 } }
I20260928 04:53:50.031374 4956 ts_tablet_manager.cc:1428] T d0e554c42ba347d19d912a7a99da7b79 P 9585c0a980e245f9833172cccbd446df: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:53:50.031424 4959 log.cc:824] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:50.031490 4673 heartbeater.cc:503] Master 127.0.98.62:46883 was elected leader, sending a full tablet report...
I20260928 04:53:50.031739 4957 raft_consensus.cc:348] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051acf36e89c4389a8a781c6603fece0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43879 } } peers { permanent_uuid: "9585c0a980e245f9833172cccbd446df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45247 } } peers { permanent_uuid: "a15a08863df049b6bb5141ed6770598b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33103 } }
I20260928 04:53:50.031903 4957 raft_consensus.cc:374] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:50.031942 4957 raft_consensus.cc:729] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15a08863df049b6bb5141ed6770598b, State: Initialized, Role: FOLLOWER
I20260928 04:53:50.032063 4957 consensus_queue.cc:260] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051acf36e89c4389a8a781c6603fece0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43879 } } peers { permanent_uuid: "9585c0a980e245f9833172cccbd446df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45247 } } peers { permanent_uuid: "a15a08863df049b6bb5141ed6770598b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33103 } }
I20260928 04:53:50.032374 4957 ts_tablet_manager.cc:1428] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:53:50.032567 4806 heartbeater.cc:503] Master 127.0.98.62:46883 was elected leader, sending a full tablet report...
I20260928 04:53:50.032600 4959 tablet_bootstrap.cc:492] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0: No bootstrap required, opened a new log
I20260928 04:53:50.032655 4959 ts_tablet_manager.cc:1397] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 04:53:50.034399 4959 raft_consensus.cc:348] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051acf36e89c4389a8a781c6603fece0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43879 } } peers { permanent_uuid: "9585c0a980e245f9833172cccbd446df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45247 } } peers { permanent_uuid: "a15a08863df049b6bb5141ed6770598b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33103 } }
I20260928 04:53:50.034607 4959 raft_consensus.cc:374] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:50.034657 4959 raft_consensus.cc:729] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 051acf36e89c4389a8a781c6603fece0, State: Initialized, Role: FOLLOWER
I20260928 04:53:50.034787 4959 consensus_queue.cc:260] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051acf36e89c4389a8a781c6603fece0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43879 } } peers { permanent_uuid: "9585c0a980e245f9833172cccbd446df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45247 } } peers { permanent_uuid: "a15a08863df049b6bb5141ed6770598b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33103 } }
I20260928 04:53:50.035086 4959 ts_tablet_manager.cc:1428] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 04:53:50.035115 4939 heartbeater.cc:503] Master 127.0.98.62:46883 was elected leader, sending a full tablet report...
W20260928 04:53:50.066442 4807 tablet.cc:2369] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:53:50.117429 4674 tablet.cc:2369] T d0e554c42ba347d19d912a7a99da7b79 P 9585c0a980e245f9833172cccbd446df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:53:50.196635 4964 raft_consensus.cc:482] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:53:50.196846 4964 raft_consensus.cc:504] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051acf36e89c4389a8a781c6603fece0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43879 } } peers { permanent_uuid: "9585c0a980e245f9833172cccbd446df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45247 } } peers { permanent_uuid: "a15a08863df049b6bb5141ed6770598b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33103 } }
I20260928 04:53:50.197250 4964 leader_election.cc:288] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9585c0a980e245f9833172cccbd446df (127.0.98.1:45247), a15a08863df049b6bb5141ed6770598b (127.0.98.2:33103)
I20260928 04:53:50.200665 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.197573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45247 (local address 127.0.98.3:58639)
0928 04:53:50.197724 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:50.197738 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:50.197837 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:50.198031 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:50.198032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:50.198034 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:50.198116 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:50.198118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.198915 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.198916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:50.199852 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:50.199854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.200394 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.200395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:50.200479 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:50.200483 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:53:50.200542 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 04:53:50.200584 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 04:53:50.200668 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.197662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58639 (local address 127.0.98.1:45247)
0928 04:53:50.197718 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:50.197720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:50.197818 (+ 98us) server_negotiation.cc:407] Connection header received
0928 04:53:50.197930 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:50.197931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:50.197944 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:50.198000 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:50.198204 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.198835 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.199949 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.200308 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.200394 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:50.200405 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:50.200555 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 04:53:50.200591 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:53:50.200793 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.197830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57973 (local address 127.0.98.2:33103)
0928 04:53:50.197884 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:50.197886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:50.198008 (+ 122us) server_negotiation.cc:407] Connection header received
0928 04:53:50.198033 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:50.198034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:50.198048 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:50.198105 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:50.198336 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.198862 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.199950 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.200425 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.200531 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:50.200540 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:50.200689 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 04:53:50.200728 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:53:50.200817 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.197721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33103 (local address 127.0.98.3:57973)
0928 04:53:50.197884 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:50.197894 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:50.197952 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:50.198113 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:50.198114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:50.198115 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:50.198227 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:50.198228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.198965 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.198965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:50.199852 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:50.199854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.200532 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.200533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:50.200611 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:50.200614 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:53:50.200683 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 04:53:50.200723 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":97,"threads_started":1}
I20260928 04:53:50.201169 4760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e554c42ba347d19d912a7a99da7b79" candidate_uuid: "051acf36e89c4389a8a781c6603fece0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15a08863df049b6bb5141ed6770598b" is_pre_election: true
I20260928 04:53:50.201200 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e554c42ba347d19d912a7a99da7b79" candidate_uuid: "051acf36e89c4389a8a781c6603fece0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9585c0a980e245f9833172cccbd446df" is_pre_election: true
I20260928 04:53:50.201362 4760 raft_consensus.cc:2454] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 051acf36e89c4389a8a781c6603fece0 in term 0.
I20260928 04:53:50.201370 4627 raft_consensus.cc:2454] T d0e554c42ba347d19d912a7a99da7b79 P 9585c0a980e245f9833172cccbd446df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 051acf36e89c4389a8a781c6603fece0 in term 0.
I20260928 04:53:50.201614 4826 leader_election.cc:302] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 051acf36e89c4389a8a781c6603fece0, 9585c0a980e245f9833172cccbd446df; no voters:
I20260928 04:53:50.201799 4964 raft_consensus.cc:2781] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:53:50.201898 4964 raft_consensus.cc:482] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:53:50.201938 4964 raft_consensus.cc:3037] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:50.202768 4964 raft_consensus.cc:504] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051acf36e89c4389a8a781c6603fece0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43879 } } peers { permanent_uuid: "9585c0a980e245f9833172cccbd446df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45247 } } peers { permanent_uuid: "a15a08863df049b6bb5141ed6770598b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33103 } }
I20260928 04:53:50.202885 4964 leader_election.cc:288] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [CANDIDATE]: Term 1 election: Requested vote from peers 9585c0a980e245f9833172cccbd446df (127.0.98.1:45247), a15a08863df049b6bb5141ed6770598b (127.0.98.2:33103)
I20260928 04:53:50.203096 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e554c42ba347d19d912a7a99da7b79" candidate_uuid: "051acf36e89c4389a8a781c6603fece0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9585c0a980e245f9833172cccbd446df"
I20260928 04:53:50.203117 4760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e554c42ba347d19d912a7a99da7b79" candidate_uuid: "051acf36e89c4389a8a781c6603fece0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15a08863df049b6bb5141ed6770598b"
I20260928 04:53:50.203195 4760 raft_consensus.cc:3037] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:50.203195 4627 raft_consensus.cc:3037] T d0e554c42ba347d19d912a7a99da7b79 P 9585c0a980e245f9833172cccbd446df [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:50.204154 4627 raft_consensus.cc:2454] T d0e554c42ba347d19d912a7a99da7b79 P 9585c0a980e245f9833172cccbd446df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 051acf36e89c4389a8a781c6603fece0 in term 1.
I20260928 04:53:50.204166 4760 raft_consensus.cc:2454] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 051acf36e89c4389a8a781c6603fece0 in term 1.
I20260928 04:53:50.204363 4826 leader_election.cc:302] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 051acf36e89c4389a8a781c6603fece0, a15a08863df049b6bb5141ed6770598b; no voters:
I20260928 04:53:50.204460 4964 raft_consensus.cc:2781] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:50.204702 4964 raft_consensus.cc:686] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [term 1 LEADER]: Becoming Leader. State: Replica: 051acf36e89c4389a8a781c6603fece0, State: Running, Role: LEADER
I20260928 04:53:50.204843 4964 consensus_queue.cc:237] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051acf36e89c4389a8a781c6603fece0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43879 } } peers { permanent_uuid: "9585c0a980e245f9833172cccbd446df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45247 } } peers { permanent_uuid: "a15a08863df049b6bb5141ed6770598b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33103 } }
I20260928 04:53:50.205766 4483 catalog_manager.cc:5495] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 051acf36e89c4389a8a781c6603fece0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "051acf36e89c4389a8a781c6603fece0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051acf36e89c4389a8a781c6603fece0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9585c0a980e245f9833172cccbd446df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a15a08863df049b6bb5141ed6770598b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33103 } health_report { overall_health: UNKNOWN } } }
I20260928 04:53:50.211659 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.209030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55928 (local address 127.0.98.3:43879)
0928 04:53:50.209095 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:50.209098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:50.209110 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:53:50.209194 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:50.209195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:50.209213 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:50.209283 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:53:50.209378 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.210011 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.210897 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:50.211093 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:50.211155 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:50.211351 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:53:50.211452 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:53:50.211527 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:50.211561 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 04:53:50.211581 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:53:50.213828 4627 raft_consensus.cc:1261] T d0e554c42ba347d19d912a7a99da7b79 P 9585c0a980e245f9833172cccbd446df [term 1 FOLLOWER]: Refusing update from remote peer 051acf36e89c4389a8a781c6603fece0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:50.213828 4760 raft_consensus.cc:1261] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b [term 1 FOLLOWER]: Refusing update from remote peer 051acf36e89c4389a8a781c6603fece0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:50.214200 4966 consensus_queue.cc:1035] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9585c0a980e245f9833172cccbd446df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:50.214350 4964 consensus_queue.cc:1035] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15a08863df049b6bb5141ed6770598b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:50.216918 4969 mvcc.cc:205] Tried to move back new op lower bound from 7334179758952255488 to 7334179758919966720. Current Snapshot: MvccSnapshot[applied={T|T < 7334179758952255488}]
I20260928 04:53:50.217065 4971 mvcc.cc:205] Tried to move back new op lower bound from 7334179758952255488 to 7334179758919966720. Current Snapshot: MvccSnapshot[applied={T|T < 7334179758952255488}]
I20260928 04:53:50.217854 4970 mvcc.cc:205] Tried to move back new op lower bound from 7334179758952255488 to 7334179758919966720. Current Snapshot: MvccSnapshot[applied={T|T < 7334179758952255488 or (T in {7334179758952255488})}]
I20260928 04:53:50.220073 4483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:53:50.220176 4483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:53:50.221040 4483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee32bac3b88645d7b28f2998e7d7eae7: Sending DeleteTablet for 3 replicas of tablet d0e554c42ba347d19d912a7a99da7b79
I20260928 04:53:50.221457 4873 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e554c42ba347d19d912a7a99da7b79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55916
I20260928 04:53:50.221473 4740 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e554c42ba347d19d912a7a99da7b79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58152
I20260928 04:53:50.221458 4607 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e554c42ba347d19d912a7a99da7b79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57982
I20260928 04:53:50.221712 4981 tablet_replica.cc:330] T d0e554c42ba347d19d912a7a99da7b79 P 9585c0a980e245f9833172cccbd446df: stopping tablet replica
I20260928 04:53:50.221828 4981 raft_consensus.cc:2229] T d0e554c42ba347d19d912a7a99da7b79 P 9585c0a980e245f9833172cccbd446df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:53:50.221959 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 4545
I20260928 04:53:50.221972 4981 raft_consensus.cc:2258] T d0e554c42ba347d19d912a7a99da7b79 P 9585c0a980e245f9833172cccbd446df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:50.227597 4979 tablet_replica.cc:330] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b: stopping tablet replica
I20260928 04:53:50.227783 4980 tablet_replica.cc:330] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0: stopping tablet replica
I20260928 04:53:50.227928 4980 raft_consensus.cc:2229] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:53:50.227955 4979 raft_consensus.cc:2229] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:53:50.228113 4980 raft_consensus.cc:2258] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:50.228240 4979 raft_consensus.cc:2258] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:50.228763 4980 ts_tablet_manager.cc:1905] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:50.228886 4979 ts_tablet_manager.cc:1905] T d0e554c42ba347d19d912a7a99da7b79 P a15a08863df049b6bb5141ed6770598b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 04:53:50.229506 4467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45247 (error 108)
I20260928 04:53:50.230207 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 4678
I20260928 04:53:50.231207 4980 ts_tablet_manager.cc:1918] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:53:50.231300 4980 log.cc:1196] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/d0e554c42ba347d19d912a7a99da7b79
I20260928 04:53:50.231627 4980 ts_tablet_manager.cc:1939] T d0e554c42ba347d19d912a7a99da7b79 P 051acf36e89c4389a8a781c6603fece0: Deleting consensus metadata
I20260928 04:53:50.232324 4467 catalog_manager.cc:4841] TS 051acf36e89c4389a8a781c6603fece0 (127.0.98.3:43879): tablet d0e554c42ba347d19d912a7a99da7b79 (table test-table [id=9296ae0879c9439782ba84e4bf9115a8]) successfully deleted
I20260928 04:53:50.232538 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.232238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45247 (local address 127.0.0.1:57984)
0928 04:53:50.232460 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:50.232506 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
W20260928 04:53:50.233585 4467 catalog_manager.cc:4584] TS 9585c0a980e245f9833172cccbd446df (127.0.98.1:45247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0e554c42ba347d19d912a7a99da7b79: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45247: connect: Connection refused (error 111)
I20260928 04:53:50.236431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 4811
I20260928 04:53:50.236533 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.236393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33103 (local address 127.0.0.1:58166)
0928 04:53:50.236448 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:50.236484 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 04:53:50.242245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 4452
2026-09-28T04:53:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2185 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T04:53:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:53:50Z Disabled control of system clock
I20260928 04:53:50.273177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37785
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 04:53:50.366159 4986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:50.366439 4986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:50.366469 4986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:50.368359 4986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:53:50.368429 4986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:50.368444 4986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:50.368459 4986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:53:50.368474 4986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:53:50.370652 4986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37785
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:50.371017 4986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:50.371367 4986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:50.377130 4994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:50.377157 4991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:50.377131 4992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:50.378167 4986 server_base.cc:1034] running on GCE node
I20260928 04:53:50.378664 4986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:50.379050 4986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:50.380223 4986 hybrid_clock.cc:648] HybridClock initialized: now 1790571230380206 us; error 38 us; skew 500 ppm
I20260928 04:53:50.381798 4986 webserver.cc:460] Webserver started at http://127.0.98.62:46409/ using document root <none> and password file <none>
I20260928 04:53:50.382088 4986 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:50.382167 4986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:50.382632 4986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:50.383878 4986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "38996df82a6348eaaf37e12ecfdaf194"
format_stamp: "Formatted at 2026-09-28 04:53:50 on dist-test-slave-m39l"
server_key: "615496262cee7beed2706a8b5203f0c5"
server_key_iv: "29991b0f80e77c6be7a76c0214b76f15"
server_key_version: "encryptionkey@0"
I20260928 04:53:50.384294 4986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "38996df82a6348eaaf37e12ecfdaf194"
format_stamp: "Formatted at 2026-09-28 04:53:50 on dist-test-slave-m39l"
server_key: "615496262cee7beed2706a8b5203f0c5"
server_key_iv: "29991b0f80e77c6be7a76c0214b76f15"
server_key_version: "encryptionkey@0"
I20260928 04:53:50.385739 4986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:53:50.386627 5000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:50.386855 4986 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:50.386967 4986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "38996df82a6348eaaf37e12ecfdaf194"
format_stamp: "Formatted at 2026-09-28 04:53:50 on dist-test-slave-m39l"
server_key: "615496262cee7beed2706a8b5203f0c5"
server_key_iv: "29991b0f80e77c6be7a76c0214b76f15"
server_key_version: "encryptionkey@0"
I20260928 04:53:50.387058 4986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:50.390825 4986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:50.391156 4986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:50.391314 4986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:50.395470 4986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37785
I20260928 04:53:50.395558 5052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37785 every 8 connection(s)
I20260928 04:53:50.395943 4986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:53:50.396715 5053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:50.398442 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 4986
I20260928 04:53:50.398634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:53:50.398873 392 external_mini_cluster.cc:1428] Setting key 4b7ebc0c06c451c4f85a40a17829daef
I20260928 04:53:50.400919 5053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194: Bootstrap starting.
I20260928 04:53:50.402544 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.399561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51068 (local address 127.0.98.62:37785)
0928 04:53:50.399978 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:50.399988 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:50.400025 (+ 37us) server_negotiation.cc:407] Connection header received
0928 04:53:50.400570 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:50.400578 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:50.400629 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:50.400765 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:50.401792 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:50.401795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:50.401796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:53:50.401800 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:50.401847 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:53:50.401858 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:50.401940 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:50.402243 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 04:53:50.402329 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":298,"threads_started":1}
I20260928 04:53:50.402678 5053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:50.403182 5053 log.cc:824] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:50.404448 5053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194: No bootstrap required, opened a new log
I20260928 04:53:50.407215 5053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38996df82a6348eaaf37e12ecfdaf194" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } }
I20260928 04:53:50.407414 5053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:50.407440 5053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38996df82a6348eaaf37e12ecfdaf194, State: Initialized, Role: FOLLOWER
I20260928 04:53:50.407584 5053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38996df82a6348eaaf37e12ecfdaf194" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } }
I20260928 04:53:50.407682 5053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:53:50.407709 5053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:53:50.407756 5053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:50.445099 5053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38996df82a6348eaaf37e12ecfdaf194" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } }
I20260928 04:53:50.445312 5053 leader_election.cc:302] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38996df82a6348eaaf37e12ecfdaf194; no voters:
I20260928 04:53:50.445571 5053 leader_election.cc:288] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:53:50.445792 5058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:50.445945 5053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:50.445966 5058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194 [term 1 LEADER]: Becoming Leader. State: Replica: 38996df82a6348eaaf37e12ecfdaf194, State: Running, Role: LEADER
I20260928 04:53:50.446071 5058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38996df82a6348eaaf37e12ecfdaf194" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } }
I20260928 04:53:50.446441 5060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38996df82a6348eaaf37e12ecfdaf194. Latest consensus state: current_term: 1 leader_uuid: "38996df82a6348eaaf37e12ecfdaf194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38996df82a6348eaaf37e12ecfdaf194" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } } }
I20260928 04:53:50.446441 5059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38996df82a6348eaaf37e12ecfdaf194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38996df82a6348eaaf37e12ecfdaf194" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } } }
I20260928 04:53:50.446694 5059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:50.446913 5060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:50.447347 5066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:53:50.448197 5066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:53:50.450997 5066 catalog_manager.cc:1347] Generated new cluster ID: b1b6ba14c6ee479cadaadedd0fe2c30f
I20260928 04:53:50.451095 5066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:53:50.480652 5066 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:53:50.480865 5066 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:53:50.495003 5066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194: Generated new TSK 0
I20260928 04:53:50.495239 5066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:53:50.508569 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37785
--builtin_ntp_servers=127.0.98.20:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 04:53:50.603364 5077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:50.603657 5077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:50.603699 5077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:50.605082 5077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:50.605178 5077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:50.605232 5077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:53:50.606933 5077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:50.607285 5077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:50.607638 5077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:50.613363 5085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:50.613408 5077 server_base.cc:1034] running on GCE node
W20260928 04:53:50.613353 5083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:50.613353 5082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:50.613834 5077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:50.614102 5077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:50.615270 5077 hybrid_clock.cc:648] HybridClock initialized: now 1790571230615253 us; error 36 us; skew 500 ppm
I20260928 04:53:50.616827 5077 webserver.cc:460] Webserver started at http://127.0.98.1:42175/ using document root <none> and password file <none>
I20260928 04:53:50.617122 5077 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:50.617187 5077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:50.617369 5077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:50.618691 5077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "1a9e187b394a4a0d9c49424a4bef702e"
format_stamp: "Formatted at 2026-09-28 04:53:50 on dist-test-slave-m39l"
server_key: "18149df3a9d321d7bb735a8b66f1d149"
server_key_iv: "7d8159fd933d63392a6f992e32387859"
server_key_version: "encryptionkey@0"
I20260928 04:53:50.619122 5077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1a9e187b394a4a0d9c49424a4bef702e"
format_stamp: "Formatted at 2026-09-28 04:53:50 on dist-test-slave-m39l"
server_key: "18149df3a9d321d7bb735a8b66f1d149"
server_key_iv: "7d8159fd933d63392a6f992e32387859"
server_key_version: "encryptionkey@0"
I20260928 04:53:50.620817 5077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:53:50.621805 5091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:50.622076 5077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:50.622205 5077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "1a9e187b394a4a0d9c49424a4bef702e"
format_stamp: "Formatted at 2026-09-28 04:53:50 on dist-test-slave-m39l"
server_key: "18149df3a9d321d7bb735a8b66f1d149"
server_key_iv: "7d8159fd933d63392a6f992e32387859"
server_key_version: "encryptionkey@0"
I20260928 04:53:50.622294 5077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:50.625943 5077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:50.626231 5077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:50.626382 5077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:50.626644 5077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:50.626977 5077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:50.627034 5077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:50.627074 5077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:50.627104 5077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:50.633222 5077 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35255
I20260928 04:53:50.633262 5204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35255 every 8 connection(s)
I20260928 04:53:50.633663 5077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:53:50.634263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 5077
I20260928 04:53:50.634421 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:53:50.634784 392 external_mini_cluster.cc:1428] Setting key 323eb7d983f90bfd915970a14cdbfb63
I20260928 04:53:50.636785 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37785
--builtin_ntp_servers=127.0.98.20:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 04:53:50.637722 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.634563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48145 (local address 127.0.98.62:37785)
0928 04:53:50.634621 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:50.634624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:50.634940 (+ 316us) server_negotiation.cc:407] Connection header received
0928 04:53:50.635707 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:50.635708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:50.635714 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:50.635774 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:50.637404 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:50.637405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:50.637405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:53:50.637409 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:50.637424 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:53:50.637428 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:50.637489 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:50.637633 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 04:53:50.637657 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:53:50.637933 5207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.634567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37785 (local address 127.0.98.1:48145)
0928 04:53:50.634796 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:50.634853 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:50.635501 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:50.635879 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:50.635884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:50.635895 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:50.637211 (+ 1316us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:53:50.637212 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:50.637259 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:53:50.637299 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:50.637499 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:50.637502 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:50.637506 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:53:50.637584 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 04:53:50.637656 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
I20260928 04:53:50.638535 5205 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37785
I20260928 04:53:50.638723 5205 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:50.638916 5205 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
I20260928 04:53:50.639544 5017 ts_manager.cc:194] Registered new tserver with Master: 1a9e187b394a4a0d9c49424a4bef702e (127.0.98.1:35255)
W20260928 04:53:50.730021 5208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:50.730268 5208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:50.730293 5208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:50.731581 5208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:50.731632 5208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:50.731662 5208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:53:50.733114 5208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:50.733402 5208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:50.733762 5208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:50.740085 5216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:50.740094 5213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:50.740139 5214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:50.740135 5208 server_base.cc:1034] running on GCE node
I20260928 04:53:50.741961 5208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:50.742282 5208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:50.743470 5208 hybrid_clock.cc:648] HybridClock initialized: now 1790571230743422 us; error 62 us; skew 500 ppm
I20260928 04:53:50.744997 5208 webserver.cc:460] Webserver started at http://127.0.98.2:43337/ using document root <none> and password file <none>
I20260928 04:53:50.745263 5208 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:50.745338 5208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:50.745491 5208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:50.746881 5208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "750ffddbe0ab4cb5bde7d5f890b52bc8"
format_stamp: "Formatted at 2026-09-28 04:53:50 on dist-test-slave-m39l"
server_key: "55f84a6b265c1b9b338de420928422bd"
server_key_iv: "1d42ae7433d671c87246dada28130f17"
server_key_version: "encryptionkey@0"
I20260928 04:53:50.747311 5208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "750ffddbe0ab4cb5bde7d5f890b52bc8"
format_stamp: "Formatted at 2026-09-28 04:53:50 on dist-test-slave-m39l"
server_key: "55f84a6b265c1b9b338de420928422bd"
server_key_iv: "1d42ae7433d671c87246dada28130f17"
server_key_version: "encryptionkey@0"
I20260928 04:53:50.748900 5208 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:53:50.749802 5222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:50.750015 5208 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:50.750177 5208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "750ffddbe0ab4cb5bde7d5f890b52bc8"
format_stamp: "Formatted at 2026-09-28 04:53:50 on dist-test-slave-m39l"
server_key: "55f84a6b265c1b9b338de420928422bd"
server_key_iv: "1d42ae7433d671c87246dada28130f17"
server_key_version: "encryptionkey@0"
I20260928 04:53:50.750272 5208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:50.753909 5208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:50.754213 5208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:50.754374 5208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:50.754631 5208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:50.754997 5208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:50.755065 5208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:50.755107 5208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:50.755136 5208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:50.761089 5208 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39229
I20260928 04:53:50.761126 5335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39229 every 8 connection(s)
I20260928 04:53:50.761516 5208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:53:50.762959 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 5208
I20260928 04:53:50.763058 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:53:50.763234 392 external_mini_cluster.cc:1428] Setting key 7fd260410c7631b119a7ce0ab8ae0897
I20260928 04:53:50.764298 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37785
--builtin_ntp_servers=127.0.98.20:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 04:53:50.765051 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.762338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41889 (local address 127.0.98.62:37785)
0928 04:53:50.762424 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:50.762426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:50.762725 (+ 299us) server_negotiation.cc:407] Connection header received
0928 04:53:50.763302 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:50.763303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:50.763312 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:50.763384 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:50.764777 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:50.764779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:50.764779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:53:50.764785 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:50.764802 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:53:50.764808 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:50.764882 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:50.764934 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 04:53:50.764949 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 04:53:50.765195 5338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.762316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37785 (local address 127.0.98.2:41889)
0928 04:53:50.762574 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:50.762618 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:50.763191 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:50.763495 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:50.763500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:50.763511 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:50.764568 (+ 1057us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:53:50.764569 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:50.764601 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:53:50.764630 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:50.764881 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:50.764884 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:50.764886 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:53:50.764941 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 04:53:50.764992 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":108,"threads_started":1}
I20260928 04:53:50.765630 5336 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37785
I20260928 04:53:50.765724 5336 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:50.765857 5336 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
I20260928 04:53:50.766222 5017 ts_manager.cc:194] Registered new tserver with Master: 750ffddbe0ab4cb5bde7d5f890b52bc8 (127.0.98.2:39229)
W20260928 04:53:50.848198 5339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:50.848469 5339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:50.848512 5339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:50.849936 5339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:50.850021 5339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:50.850072 5339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:53:50.851733 5339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:50.852065 5339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:50.852435 5339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:50.858238 5345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:50.858260 5344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:50.858240 5347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:50.858263 5339 server_base.cc:1034] running on GCE node
I20260928 04:53:50.858820 5339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:50.859361 5339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:50.860610 5339 hybrid_clock.cc:648] HybridClock initialized: now 1790571230860585 us; error 40 us; skew 500 ppm
I20260928 04:53:50.862119 5339 webserver.cc:460] Webserver started at http://127.0.98.3:40137/ using document root <none> and password file <none>
I20260928 04:53:50.862401 5339 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:50.862452 5339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:50.862618 5339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:50.863857 5339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "75b7d9ae063c459da787c3316472359c"
format_stamp: "Formatted at 2026-09-28 04:53:50 on dist-test-slave-m39l"
server_key: "842e0e192f0a359a2f26b009d359f64e"
server_key_iv: "bf8dd4d1217e4fdce09f68e97291f923"
server_key_version: "encryptionkey@0"
I20260928 04:53:50.864259 5339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "75b7d9ae063c459da787c3316472359c"
format_stamp: "Formatted at 2026-09-28 04:53:50 on dist-test-slave-m39l"
server_key: "842e0e192f0a359a2f26b009d359f64e"
server_key_iv: "bf8dd4d1217e4fdce09f68e97291f923"
server_key_version: "encryptionkey@0"
I20260928 04:53:50.865860 5339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:53:50.866846 5353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:50.867087 5339 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:50.867202 5339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "75b7d9ae063c459da787c3316472359c"
format_stamp: "Formatted at 2026-09-28 04:53:50 on dist-test-slave-m39l"
server_key: "842e0e192f0a359a2f26b009d359f64e"
server_key_iv: "bf8dd4d1217e4fdce09f68e97291f923"
server_key_version: "encryptionkey@0"
I20260928 04:53:50.867282 5339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:50.871971 5339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:50.872325 5339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:50.872498 5339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:50.872758 5339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:50.873116 5339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:50.873178 5339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:50.873227 5339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:50.873255 5339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:50.880045 5339 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43303
I20260928 04:53:50.880092 5466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43303 every 8 connection(s)
I20260928 04:53:50.880498 5339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:53:50.883885 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.881370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44809 (local address 127.0.98.62:37785)
0928 04:53:50.881462 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:50.881464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:50.881689 (+ 225us) server_negotiation.cc:407] Connection header received
0928 04:53:50.882197 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:50.882198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:50.882204 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:50.882263 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:50.883641 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:50.883643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:50.883643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:53:50.883649 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:50.883666 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:53:50.883670 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:50.883740 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:50.883816 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 04:53:50.883830 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 04:53:50.884089 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.881332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37785 (local address 127.0.98.3:44809)
0928 04:53:50.881533 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:50.881572 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:50.882079 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:50.882322 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:50.882325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:50.882334 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:50.883461 (+ 1127us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:53:50.883462 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:50.883494 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:53:50.883528 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:50.883744 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:50.883747 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:50.883749 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:53:50.883810 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 04:53:50.883865 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":108,"threads_started":1}
I20260928 04:53:50.884572 5467 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37785
I20260928 04:53:50.884671 5467 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:50.884794 5467 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
I20260928 04:53:50.885200 5017 ts_manager.cc:194] Registered new tserver with Master: 75b7d9ae063c459da787c3316472359c (127.0.98.3:43303)
I20260928 04:53:50.889806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 5339
I20260928 04:53:50.889946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:53:50.890127 392 external_mini_cluster.cc:1428] Setting key ae04243305201fb0050c9a23f973dc64
I20260928 04:53:50.891119 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:53:50.893728 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.892512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51074 (local address 127.0.98.62:37785)
0928 04:53:50.892586 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:50.892589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:50.892693 (+ 104us) server_negotiation.cc:407] Connection header received
0928 04:53:50.892739 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:50.892740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:50.892745 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:50.892799 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:50.893382 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:50.893383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:50.893383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:53:50.893387 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:50.893401 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:53:50.893404 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:50.893475 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:50.893614 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 04:53:50.893643 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 04:53:50.896564 5017 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:53:50.898278 5017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:53:50.903021 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.900992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43303 (local address 127.0.0.1:57782)
0928 04:53:50.901746 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:50.901758 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:50.901821 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:50.902040 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:50.902041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:50.902044 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:50.902592 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:53:50.902593 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:50.902611 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:53:50.902638 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:50.902843 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:50.902847 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:50.902851 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:53:50.902921 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:53:50.902936 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":609,"thread_start_us":83,"threads_started":1}
I20260928 04:53:50.903052 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.901122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.0.98.3:43303)
0928 04:53:50.901336 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:50.901341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:50.901880 (+ 539us) server_negotiation.cc:407] Connection header received
0928 04:53:50.901908 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:50.901911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:50.901921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:50.902006 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:50.902707 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:50.902714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:50.902715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:53:50.902720 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:50.902759 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:53:50.902787 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:50.902853 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:50.902930 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 04:53:50.902946 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":111,"threads_started":1}
I20260928 04:53:50.903335 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.900992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39229 (local address 127.0.0.1:37568)
0928 04:53:50.901132 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:50.901176 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:50.901274 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:50.901624 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:50.901629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:50.901637 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:50.902895 (+ 1258us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:53:50.902896 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:50.902902 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:53:50.902914 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:50.903173 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:50.903174 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:50.903175 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 04:53:50.903247 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 04:53:50.903261 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260928 04:53:50.903345 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.901307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37568 (local address 127.0.98.2:39229)
0928 04:53:50.901478 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:50.901481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:50.901489 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:53:50.901513 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:50.901515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:50.901523 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:50.901606 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:50.902996 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:50.903005 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:50.903005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:53:50.903011 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:50.903045 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:53:50.903073 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:50.903165 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:50.903254 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 04:53:50.903272 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
I20260928 04:53:50.903896 5401 tablet_service.cc:1467] Processing CreateTablet for tablet 0412be5e0e66408f86a1b19be5e8741e (DEFAULT_TABLE table=test-table [id=ab8bf803c47841208c572d6cd4a092f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:50.904296 5401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0412be5e0e66408f86a1b19be5e8741e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:50.904603 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.900992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35255 (local address 127.0.0.1:39716)
0928 04:53:50.901747 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:50.901758 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:50.902468 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:50.903038 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:50.903039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:50.903042 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:50.904161 (+ 1119us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:53:50.904161 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:50.904168 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:53:50.904182 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:50.904462 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:50.904463 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:50.904464 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 04:53:50.904526 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 04:53:50.904539 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":2,"thread_start_us":73,"threads_started":1}
I20260928 04:53:50.904641 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.901462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.0.98.1:35255)
0928 04:53:50.901620 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:50.901625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:50.901821 (+ 196us) server_negotiation.cc:407] Connection header received
0928 04:53:50.902584 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:50.902593 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:50.902604 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:50.902725 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:50.904250 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:50.904260 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:50.904260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:53:50.904283 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:50.904317 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:53:50.904364 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:50.904467 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:50.904530 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 04:53:50.904547 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260928 04:53:50.905331 5139 tablet_service.cc:1467] Processing CreateTablet for tablet 0412be5e0e66408f86a1b19be5e8741e (DEFAULT_TABLE table=test-table [id=ab8bf803c47841208c572d6cd4a092f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:50.905669 5139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0412be5e0e66408f86a1b19be5e8741e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:50.906988 5270 tablet_service.cc:1467] Processing CreateTablet for tablet 0412be5e0e66408f86a1b19be5e8741e (DEFAULT_TABLE table=test-table [id=ab8bf803c47841208c572d6cd4a092f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:50.907337 5270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0412be5e0e66408f86a1b19be5e8741e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:50.907449 5481 tablet_bootstrap.cc:492] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c: Bootstrap starting.
I20260928 04:53:50.908083 5481 tablet_bootstrap.cc:654] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:50.908394 5481 log.cc:824] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:50.908772 5482 tablet_bootstrap.cc:492] T 0412be5e0e66408f86a1b19be5e8741e P 1a9e187b394a4a0d9c49424a4bef702e: Bootstrap starting.
I20260928 04:53:50.909475 5482 tablet_bootstrap.cc:654] T 0412be5e0e66408f86a1b19be5e8741e P 1a9e187b394a4a0d9c49424a4bef702e: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:50.909778 5482 log.cc:824] T 0412be5e0e66408f86a1b19be5e8741e P 1a9e187b394a4a0d9c49424a4bef702e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:50.910338 5485 tablet_bootstrap.cc:492] T 0412be5e0e66408f86a1b19be5e8741e P 750ffddbe0ab4cb5bde7d5f890b52bc8: Bootstrap starting.
I20260928 04:53:50.910657 5482 tablet_bootstrap.cc:492] T 0412be5e0e66408f86a1b19be5e8741e P 1a9e187b394a4a0d9c49424a4bef702e: No bootstrap required, opened a new log
I20260928 04:53:50.910717 5482 ts_tablet_manager.cc:1397] T 0412be5e0e66408f86a1b19be5e8741e P 1a9e187b394a4a0d9c49424a4bef702e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:53:50.910990 5485 tablet_bootstrap.cc:654] T 0412be5e0e66408f86a1b19be5e8741e P 750ffddbe0ab4cb5bde7d5f890b52bc8: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:50.911298 5485 log.cc:824] T 0412be5e0e66408f86a1b19be5e8741e P 750ffddbe0ab4cb5bde7d5f890b52bc8: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:50.912101 5485 tablet_bootstrap.cc:492] T 0412be5e0e66408f86a1b19be5e8741e P 750ffddbe0ab4cb5bde7d5f890b52bc8: No bootstrap required, opened a new log
I20260928 04:53:50.912189 5485 ts_tablet_manager.cc:1397] T 0412be5e0e66408f86a1b19be5e8741e P 750ffddbe0ab4cb5bde7d5f890b52bc8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:53:50.913111 5482 raft_consensus.cc:348] T 0412be5e0e66408f86a1b19be5e8741e P 1a9e187b394a4a0d9c49424a4bef702e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9e187b394a4a0d9c49424a4bef702e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35255 } } peers { permanent_uuid: "75b7d9ae063c459da787c3316472359c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43303 } } peers { permanent_uuid: "750ffddbe0ab4cb5bde7d5f890b52bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39229 } }
I20260928 04:53:50.913272 5482 raft_consensus.cc:374] T 0412be5e0e66408f86a1b19be5e8741e P 1a9e187b394a4a0d9c49424a4bef702e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:50.913295 5482 raft_consensus.cc:729] T 0412be5e0e66408f86a1b19be5e8741e P 1a9e187b394a4a0d9c49424a4bef702e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9e187b394a4a0d9c49424a4bef702e, State: Initialized, Role: FOLLOWER
I20260928 04:53:50.913429 5482 consensus_queue.cc:260] T 0412be5e0e66408f86a1b19be5e8741e P 1a9e187b394a4a0d9c49424a4bef702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9e187b394a4a0d9c49424a4bef702e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35255 } } peers { permanent_uuid: "75b7d9ae063c459da787c3316472359c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43303 } } peers { permanent_uuid: "750ffddbe0ab4cb5bde7d5f890b52bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39229 } }
I20260928 04:53:50.913712 5482 ts_tablet_manager.cc:1428] T 0412be5e0e66408f86a1b19be5e8741e P 1a9e187b394a4a0d9c49424a4bef702e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:53:50.913764 5205 heartbeater.cc:503] Master 127.0.98.62:37785 was elected leader, sending a full tablet report...
I20260928 04:53:50.914206 5481 tablet_bootstrap.cc:492] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c: No bootstrap required, opened a new log
I20260928 04:53:50.914288 5481 ts_tablet_manager.cc:1397] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 04:53:50.914426 5485 raft_consensus.cc:348] T 0412be5e0e66408f86a1b19be5e8741e P 750ffddbe0ab4cb5bde7d5f890b52bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9e187b394a4a0d9c49424a4bef702e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35255 } } peers { permanent_uuid: "75b7d9ae063c459da787c3316472359c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43303 } } peers { permanent_uuid: "750ffddbe0ab4cb5bde7d5f890b52bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39229 } }
I20260928 04:53:50.914623 5485 raft_consensus.cc:374] T 0412be5e0e66408f86a1b19be5e8741e P 750ffddbe0ab4cb5bde7d5f890b52bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:50.914680 5485 raft_consensus.cc:729] T 0412be5e0e66408f86a1b19be5e8741e P 750ffddbe0ab4cb5bde7d5f890b52bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750ffddbe0ab4cb5bde7d5f890b52bc8, State: Initialized, Role: FOLLOWER
I20260928 04:53:50.914815 5485 consensus_queue.cc:260] T 0412be5e0e66408f86a1b19be5e8741e P 750ffddbe0ab4cb5bde7d5f890b52bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9e187b394a4a0d9c49424a4bef702e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35255 } } peers { permanent_uuid: "75b7d9ae063c459da787c3316472359c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43303 } } peers { permanent_uuid: "750ffddbe0ab4cb5bde7d5f890b52bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39229 } }
I20260928 04:53:50.915118 5485 ts_tablet_manager.cc:1428] T 0412be5e0e66408f86a1b19be5e8741e P 750ffddbe0ab4cb5bde7d5f890b52bc8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:53:50.915141 5336 heartbeater.cc:503] Master 127.0.98.62:37785 was elected leader, sending a full tablet report...
I20260928 04:53:50.916879 5481 raft_consensus.cc:348] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9e187b394a4a0d9c49424a4bef702e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35255 } } peers { permanent_uuid: "75b7d9ae063c459da787c3316472359c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43303 } } peers { permanent_uuid: "750ffddbe0ab4cb5bde7d5f890b52bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39229 } }
I20260928 04:53:50.917047 5481 raft_consensus.cc:374] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:50.917093 5481 raft_consensus.cc:729] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b7d9ae063c459da787c3316472359c, State: Initialized, Role: FOLLOWER
I20260928 04:53:50.917220 5481 consensus_queue.cc:260] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9e187b394a4a0d9c49424a4bef702e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35255 } } peers { permanent_uuid: "75b7d9ae063c459da787c3316472359c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43303 } } peers { permanent_uuid: "750ffddbe0ab4cb5bde7d5f890b52bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39229 } }
I20260928 04:53:50.917481 5481 ts_tablet_manager.cc:1428] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 04:53:50.917579 5467 heartbeater.cc:503] Master 127.0.98.62:37785 was elected leader, sending a full tablet report...
I20260928 04:53:50.919468 5489 raft_consensus.cc:482] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:53:50.919586 5489 raft_consensus.cc:504] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9e187b394a4a0d9c49424a4bef702e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35255 } } peers { permanent_uuid: "75b7d9ae063c459da787c3316472359c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43303 } } peers { permanent_uuid: "750ffddbe0ab4cb5bde7d5f890b52bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39229 } }
I20260928 04:53:50.919966 5489 leader_election.cc:288] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9e187b394a4a0d9c49424a4bef702e (127.0.98.1:35255), 750ffddbe0ab4cb5bde7d5f890b52bc8 (127.0.98.2:39229)
I20260928 04:53:50.921489 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.920205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40601 (local address 127.0.98.1:35255)
0928 04:53:50.920255 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:50.920257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:50.920348 (+ 91us) server_negotiation.cc:407] Connection header received
0928 04:53:50.920412 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:50.920413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:50.920420 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:50.920494 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:50.921249 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:50.921251 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:50.921251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:53:50.921256 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:50.921272 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:53:50.921276 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:50.921352 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:50.921408 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 04:53:50.921422 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:53:50.921586 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.920250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48613 (local address 127.0.98.2:39229)
0928 04:53:50.920294 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:50.920296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:50.920413 (+ 117us) server_negotiation.cc:407] Connection header received
0928 04:53:50.920437 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:50.920438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:50.920443 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:50.920495 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:50.921356 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:50.921358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:50.921358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:53:50.921363 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:50.921380 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:53:50.921384 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:50.921451 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:50.921501 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 04:53:50.921515 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:53:50.921593 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.920134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39229 (local address 127.0.98.3:48613)
0928 04:53:50.920306 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:50.920318 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:50.920359 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:50.920535 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:50.920536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:50.920538 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:50.921230 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:53:50.921231 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:50.921237 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:53:50.921248 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:50.921444 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:50.921445 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:50.921445 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 04:53:50.921498 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 04:53:50.921512 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":109,"threads_started":1}
I20260928 04:53:50.921505 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.920105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35255 (local address 127.0.98.3:40601)
0928 04:53:50.920290 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:50.920303 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:50.920358 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:50.920497 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:50.920497 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:50.920500 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:50.921119 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:53:50.921120 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:50.921128 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:53:50.921143 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:50.921349 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:50.921350 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:50.921351 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 04:53:50.921406 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 04:53:50.921421 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260928 04:53:50.921972 5290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0412be5e0e66408f86a1b19be5e8741e" candidate_uuid: "75b7d9ae063c459da787c3316472359c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750ffddbe0ab4cb5bde7d5f890b52bc8" is_pre_election: true
I20260928 04:53:50.922155 5290 raft_consensus.cc:2454] T 0412be5e0e66408f86a1b19be5e8741e P 750ffddbe0ab4cb5bde7d5f890b52bc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75b7d9ae063c459da787c3316472359c in term 0.
I20260928 04:53:50.922147 5159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0412be5e0e66408f86a1b19be5e8741e" candidate_uuid: "75b7d9ae063c459da787c3316472359c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9e187b394a4a0d9c49424a4bef702e" is_pre_election: true
I20260928 04:53:50.922303 5159 raft_consensus.cc:2454] T 0412be5e0e66408f86a1b19be5e8741e P 1a9e187b394a4a0d9c49424a4bef702e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75b7d9ae063c459da787c3316472359c in term 0.
I20260928 04:53:50.922418 5354 leader_election.cc:302] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 750ffddbe0ab4cb5bde7d5f890b52bc8, 75b7d9ae063c459da787c3316472359c; no voters:
I20260928 04:53:50.922592 5489 raft_consensus.cc:2781] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:53:50.922679 5489 raft_consensus.cc:482] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:53:50.922716 5489 raft_consensus.cc:3037] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:50.947180 5489 raft_consensus.cc:504] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9e187b394a4a0d9c49424a4bef702e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35255 } } peers { permanent_uuid: "75b7d9ae063c459da787c3316472359c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43303 } } peers { permanent_uuid: "750ffddbe0ab4cb5bde7d5f890b52bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39229 } }
I20260928 04:53:50.947444 5489 leader_election.cc:288] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [CANDIDATE]: Term 1 election: Requested vote from peers 1a9e187b394a4a0d9c49424a4bef702e (127.0.98.1:35255), 750ffddbe0ab4cb5bde7d5f890b52bc8 (127.0.98.2:39229)
I20260928 04:53:50.947743 5290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0412be5e0e66408f86a1b19be5e8741e" candidate_uuid: "75b7d9ae063c459da787c3316472359c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750ffddbe0ab4cb5bde7d5f890b52bc8"
I20260928 04:53:50.947760 5159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0412be5e0e66408f86a1b19be5e8741e" candidate_uuid: "75b7d9ae063c459da787c3316472359c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9e187b394a4a0d9c49424a4bef702e"
I20260928 04:53:50.947860 5290 raft_consensus.cc:3037] T 0412be5e0e66408f86a1b19be5e8741e P 750ffddbe0ab4cb5bde7d5f890b52bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:50.947860 5159 raft_consensus.cc:3037] T 0412be5e0e66408f86a1b19be5e8741e P 1a9e187b394a4a0d9c49424a4bef702e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:50.948927 5290 raft_consensus.cc:2454] T 0412be5e0e66408f86a1b19be5e8741e P 750ffddbe0ab4cb5bde7d5f890b52bc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b7d9ae063c459da787c3316472359c in term 1.
I20260928 04:53:50.948915 5159 raft_consensus.cc:2454] T 0412be5e0e66408f86a1b19be5e8741e P 1a9e187b394a4a0d9c49424a4bef702e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b7d9ae063c459da787c3316472359c in term 1.
I20260928 04:53:50.949187 5354 leader_election.cc:302] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 750ffddbe0ab4cb5bde7d5f890b52bc8, 75b7d9ae063c459da787c3316472359c; no voters:
I20260928 04:53:50.949344 5489 raft_consensus.cc:2781] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:50.949527 5489 raft_consensus.cc:686] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [term 1 LEADER]: Becoming Leader. State: Replica: 75b7d9ae063c459da787c3316472359c, State: Running, Role: LEADER
I20260928 04:53:50.949657 5489 consensus_queue.cc:237] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9e187b394a4a0d9c49424a4bef702e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35255 } } peers { permanent_uuid: "75b7d9ae063c459da787c3316472359c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43303 } } peers { permanent_uuid: "750ffddbe0ab4cb5bde7d5f890b52bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39229 } }
I20260928 04:53:50.950690 5017 catalog_manager.cc:5495] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c reported cstate change: term changed from 0 to 1, leader changed from <none> to 75b7d9ae063c459da787c3316472359c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "75b7d9ae063c459da787c3316472359c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9e187b394a4a0d9c49424a4bef702e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75b7d9ae063c459da787c3316472359c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "750ffddbe0ab4cb5bde7d5f890b52bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39229 } health_report { overall_health: UNKNOWN } } }
I20260928 04:53:50.965703 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.964617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.0.98.3:43303)
0928 04:53:50.964686 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:50.964688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:50.964697 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:53:50.964736 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:50.964736 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:50.964749 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:50.964818 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:50.965443 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:50.965444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:50.965445 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:53:50.965449 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:50.965478 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:53:50.965483 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:50.965552 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:50.965609 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 04:53:50.965623 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:53:50.967789 5290 raft_consensus.cc:1261] T 0412be5e0e66408f86a1b19be5e8741e P 750ffddbe0ab4cb5bde7d5f890b52bc8 [term 1 FOLLOWER]: Refusing update from remote peer 75b7d9ae063c459da787c3316472359c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:50.967816 5159 raft_consensus.cc:1261] T 0412be5e0e66408f86a1b19be5e8741e P 1a9e187b394a4a0d9c49424a4bef702e [term 1 FOLLOWER]: Refusing update from remote peer 75b7d9ae063c459da787c3316472359c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:50.968217 5489 consensus_queue.cc:1035] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [LEADER]: Connected to new peer: Peer: permanent_uuid: "750ffddbe0ab4cb5bde7d5f890b52bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:50.968441 5489 consensus_queue.cc:1035] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9e187b394a4a0d9c49424a4bef702e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:50.971109 5494 mvcc.cc:205] Tried to move back new op lower bound from 7334179762040963072 to 7334179761970737152. Current Snapshot: MvccSnapshot[applied={T|T < 7334179762040963072}]
I20260928 04:53:50.971664 5496 mvcc.cc:205] Tried to move back new op lower bound from 7334179762040963072 to 7334179761970737152. Current Snapshot: MvccSnapshot[applied={T|T < 7334179762040963072}]
I20260928 04:53:50.971939 5495 mvcc.cc:205] Tried to move back new op lower bound from 7334179762040963072 to 7334179761970737152. Current Snapshot: MvccSnapshot[applied={T|T < 7334179762040963072}]
I20260928 04:53:50.973488 5017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:53:50.973595 5017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:53:50.974481 5017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38996df82a6348eaaf37e12ecfdaf194: Sending DeleteTablet for 3 replicas of tablet 0412be5e0e66408f86a1b19be5e8741e
I20260928 04:53:50.974876 5139 tablet_service.cc:1514] Processing DeleteTablet for tablet 0412be5e0e66408f86a1b19be5e8741e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:50 UTC) from {username='slave'} at 127.0.0.1:39716
I20260928 04:53:50.974958 5270 tablet_service.cc:1514] Processing DeleteTablet for tablet 0412be5e0e66408f86a1b19be5e8741e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:50 UTC) from {username='slave'} at 127.0.0.1:37568
I20260928 04:53:50.974959 5401 tablet_service.cc:1514] Processing DeleteTablet for tablet 0412be5e0e66408f86a1b19be5e8741e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:50 UTC) from {username='slave'} at 127.0.0.1:57782
I20260928 04:53:50.975236 5505 tablet_replica.cc:330] T 0412be5e0e66408f86a1b19be5e8741e P 750ffddbe0ab4cb5bde7d5f890b52bc8: stopping tablet replica
I20260928 04:53:50.975351 5505 raft_consensus.cc:2229] T 0412be5e0e66408f86a1b19be5e8741e P 750ffddbe0ab4cb5bde7d5f890b52bc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:53:50.975430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 5077
I20260928 04:53:50.975474 5505 raft_consensus.cc:2258] T 0412be5e0e66408f86a1b19be5e8741e P 750ffddbe0ab4cb5bde7d5f890b52bc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:50.975736 5506 tablet_replica.cc:330] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c: stopping tablet replica
I20260928 04:53:50.975827 5506 raft_consensus.cc:2229] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:53:50.975916 5506 raft_consensus.cc:2258] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:50.976361 5506 ts_tablet_manager.cc:1905] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:50.983105 5505 ts_tablet_manager.cc:1905] T 0412be5e0e66408f86a1b19be5e8741e P 750ffddbe0ab4cb5bde7d5f890b52bc8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 04:53:50.983582 5003 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35255 (error 108)
I20260928 04:53:50.983827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 5208
I20260928 04:53:50.986071 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:50.985873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35255 (local address 127.0.0.1:39726)
0928 04:53:50.985958 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:50.986024 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 04:53:50.986274 5003 catalog_manager.cc:4584] TS 1a9e187b394a4a0d9c49424a4bef702e (127.0.98.1:35255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0412be5e0e66408f86a1b19be5e8741e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35255: connect: Connection refused (error 111)
I20260928 04:53:51.039283 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:51.039039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35255 (local address 127.0.0.1:39730)
0928 04:53:51.039166 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:51.039225 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 04:53:51.039727 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:51.039614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35255 (local address 127.0.0.1:39736)
0928 04:53:51.039664 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:51.039698 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 04:53:51.042456 5506 ts_tablet_manager.cc:1918] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:53:51.042588 5506 log.cc:1196] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/0412be5e0e66408f86a1b19be5e8741e
I20260928 04:53:51.042879 5506 ts_tablet_manager.cc:1939] T 0412be5e0e66408f86a1b19be5e8741e P 75b7d9ae063c459da787c3316472359c: Deleting consensus metadata
I20260928 04:53:51.045632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 5339
I20260928 04:53:51.045627 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:51.045504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39229 (local address 127.0.0.1:37572)
0928 04:53:51.045553 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:51.045579 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 04:53:51.067500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 4986
I20260928 04:53:51.067770 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:51.067535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43303 (local address 127.0.0.1:57798)
0928 04:53:51.067627 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:51.067668 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
2026-09-28T04:53:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (823 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 04:53:51.087973 392 mini_oidc.cc:152] Starting JWKS server
I20260928 04:53:51.089092 392 webserver.cc:460] Webserver started at https://127.0.0.1:43593/ using document root <none> and password file <none>
I20260928 04:53:51.089174 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 04:53:51.089301 392 webserver.cc:460] Webserver started at http://127.0.0.1:41845/ using document root <none> and password file <none>
2026-09-28T04:53:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:53:51Z Disabled control of system clock
I20260928 04:53:51.097803 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42107
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42107
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43593/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/testchainca.cert with env {}
W20260928 04:53:51.182830 5513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:51.183060 5513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:51.183084 5513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:51.184362 5513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:53:51.184414 5513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:51.184427 5513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:51.184440 5513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:53:51.184453 5513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 04:53:51.184465 5513 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 04:53:51.184476 5513 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43593/jwks.json
I20260928 04:53:51.186151 5513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42107
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43593/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:51.186379 5513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:51.186590 5513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:51.191977 5519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:51.191962 5518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:51.191962 5521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:51.192324 5513 server_base.cc:1034] running on GCE node
I20260928 04:53:51.192606 5513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:51.192912 5513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:51.194096 5513 hybrid_clock.cc:648] HybridClock initialized: now 1790571231194071 us; error 30 us; skew 500 ppm
I20260928 04:53:51.195729 5513 webserver.cc:460] Webserver started at http://127.0.98.62:40767/ using document root <none> and password file <none>
I20260928 04:53:51.196040 5513 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:51.196097 5513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:51.196242 5513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:51.197412 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "e1278fdd79754c28bea39776d5c4f658"
format_stamp: "Formatted at 2026-09-28 04:53:51 on dist-test-slave-m39l"
server_key: "7ca2a1b455c1bc5603d5c14c84d2d7ea"
server_key_iv: "3a44b175d768892cc7b3f9b1215eb805"
server_key_version: "encryptionkey@0"
I20260928 04:53:51.197821 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "e1278fdd79754c28bea39776d5c4f658"
format_stamp: "Formatted at 2026-09-28 04:53:51 on dist-test-slave-m39l"
server_key: "7ca2a1b455c1bc5603d5c14c84d2d7ea"
server_key_iv: "3a44b175d768892cc7b3f9b1215eb805"
server_key_version: "encryptionkey@0"
I20260928 04:53:51.199390 5513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:53:51.200482 5527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:51.200793 5513 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:51.200917 5513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "e1278fdd79754c28bea39776d5c4f658"
format_stamp: "Formatted at 2026-09-28 04:53:51 on dist-test-slave-m39l"
server_key: "7ca2a1b455c1bc5603d5c14c84d2d7ea"
server_key_iv: "3a44b175d768892cc7b3f9b1215eb805"
server_key_version: "encryptionkey@0"
I20260928 04:53:51.201006 5513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:51.221237 5513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:51.221603 5513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:51.221776 5513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:51.226078 5513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42107
I20260928 04:53:51.226130 5579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42107 every 8 connection(s)
I20260928 04:53:51.226537 5513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:53:51.227588 5580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:51.230293 5580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658: Bootstrap starting.
I20260928 04:53:51.230989 5580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:51.231315 5580 log.cc:824] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:51.232077 5580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658: No bootstrap required, opened a new log
I20260928 04:53:51.232851 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 5513
I20260928 04:53:51.232972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:53:51.233165 392 external_mini_cluster.cc:1428] Setting key 56888b9e7feb967c29ffeb66aef8fdc0
I20260928 04:53:51.234413 5580 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1278fdd79754c28bea39776d5c4f658" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42107 } }
I20260928 04:53:51.234630 5580 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:51.234671 5580 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1278fdd79754c28bea39776d5c4f658, State: Initialized, Role: FOLLOWER
I20260928 04:53:51.234813 5580 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1278fdd79754c28bea39776d5c4f658" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42107 } }
I20260928 04:53:51.234902 5580 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:53:51.234932 5580 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:53:51.234972 5580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:51.238571 5580 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1278fdd79754c28bea39776d5c4f658" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42107 } }
I20260928 04:53:51.238725 5580 leader_election.cc:302] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1278fdd79754c28bea39776d5c4f658; no voters:
I20260928 04:53:51.238929 5580 leader_election.cc:288] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:53:51.239014 5585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:51.239173 5585 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658 [term 1 LEADER]: Becoming Leader. State: Replica: e1278fdd79754c28bea39776d5c4f658, State: Running, Role: LEADER
I20260928 04:53:51.239262 5580 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:51.239503 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:51.233755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37680 (local address 127.0.98.62:42107)
0928 04:53:51.233985 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:51.233994 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:51.234031 (+ 37us) server_negotiation.cc:407] Connection header received
0928 04:53:51.234369 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:51.234378 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:51.234462 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:51.234644 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:51.234896 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:51.236346 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:51.237128 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:51.237533 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:51.237680 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:51.238283 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:51.238286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:51.238287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:53:51.238592 (+ 305us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:51.238641 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:53:51.238658 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:51.238732 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:51.239041 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 04:53:51.239154 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260928 04:53:51.239274 5585 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1278fdd79754c28bea39776d5c4f658" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42107 } }
I20260928 04:53:51.240499 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:53:51.240411 5585 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1278fdd79754c28bea39776d5c4f658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1278fdd79754c28bea39776d5c4f658" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42107 } } }
I20260928 04:53:51.242786 5585 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658 [sys.catalog]: This master's current role is: LEADER
W20260928 04:53:51.242033 5599 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:53:51.242940 5599 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:53:51.242164 5587 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1278fdd79754c28bea39776d5c4f658. Latest consensus state: current_term: 1 leader_uuid: "e1278fdd79754c28bea39776d5c4f658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1278fdd79754c28bea39776d5c4f658" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42107 } } }
I20260928 04:53:51.242995 5587 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:51.244302 5602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:53:51.244530 5602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:53:51.246096 5602 catalog_manager.cc:1347] Generated new cluster ID: 4e7445ec67034d8c9b004b50f24bae94
I20260928 04:53:51.246152 5602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:53:51.257424 5602 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:53:51.258095 5602 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:53:51.261742 5602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1278fdd79754c28bea39776d5c4f658: Generated new TSK 0
I20260928 04:53:51.261951 5602 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 04:53:51.294915 5610 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 04:53:51.295032 5610 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 04:53:51.295182 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:51.270697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37684 (local address 127.0.98.62:42107)
0928 04:53:51.270766 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:51.270769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:51.270889 (+ 120us) server_negotiation.cc:407] Connection header received
0928 04:53:51.270969 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:51.270970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:51.270984 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:51.271065 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0928 04:53:51.271248 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:51.272041 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:51.273284 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:51.273476 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:51.273538 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:51.273810 (+ 272us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 04:53:51.294869 (+ 21059us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43593/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 04:53:51.295090 (+ 221us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37684: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43593/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 04:53:51.295244 5584 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37684: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43593/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 04:53:51.295603 5609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:53:51.270617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42107 (local address 127.0.0.1:37684)
0928 04:53:51.270820 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:51.270830 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:51.270890 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:51.271082 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:51.271083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:51.271085 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 04:53:51.271176 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:51.271178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:51.272161 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:51.272162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:51.273180 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:51.273183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:51.273560 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:51.273564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:51.273644 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:51.273678 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 04:53:51.295053 (+ 21375us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43593/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 04:53:51.295186 (+ 133us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42107: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43593/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":106,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:42107: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43593/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 04:53:51.296203 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 5513
2026-09-28T04:53:51Z chronyd exiting
I20260928 04:53:51.332721 392 test_util.cc:175] -----------------------------------------------
I20260928 04:53:51.332818 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790571208412331-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (245 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 04:53:51.333825 392 mini_oidc.cc:152] Starting JWKS server
I20260928 04:53:51.334697 392 webserver.cc:460] Webserver started at https://127.0.0.1:36037/ using document root <none> and password file <none>
I20260928 04:53:51.334792 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 04:53:51.334975 392 webserver.cc:460] Webserver started at http://127.0.0.1:37279/ using document root <none> and password file <none>
2026-09-28T04:53:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:53:51Z Disabled control of system clock
I20260928 04:53:51.343416 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43051
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43051
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36037/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/testchainca.cert with env {}
W20260928 04:53:51.429314 5616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:51.429600 5616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:51.429695 5616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:51.431044 5616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:53:51.431119 5616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:51.431140 5616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:51.431161 5616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:53:51.431200 5616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 04:53:51.431223 5616 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 04:53:51.431242 5616 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36037/jwks.json
I20260928 04:53:51.432976 5616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43051
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36037/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:51.433310 5616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:51.433552 5616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:51.439000 5622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:51.439023 5621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:51.439073 5624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:51.439311 5616 server_base.cc:1034] running on GCE node
I20260928 04:53:51.439579 5616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:51.439891 5616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:51.441059 5616 hybrid_clock.cc:648] HybridClock initialized: now 1790571231441033 us; error 54 us; skew 500 ppm
I20260928 04:53:51.442480 5616 webserver.cc:460] Webserver started at http://127.0.98.62:35075/ using document root <none> and password file <none>
I20260928 04:53:51.442811 5616 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:51.442888 5616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:51.443038 5616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:51.444162 5616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "879ea20f9b8042f9bc553dbfaf749286"
format_stamp: "Formatted at 2026-09-28 04:53:51 on dist-test-slave-m39l"
server_key: "13dc8ece277bc8236af150021ea31ebc"
server_key_iv: "d1b1294264a8ee54c7156dab5793b931"
server_key_version: "encryptionkey@0"
I20260928 04:53:51.444567 5616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "879ea20f9b8042f9bc553dbfaf749286"
format_stamp: "Formatted at 2026-09-28 04:53:51 on dist-test-slave-m39l"
server_key: "13dc8ece277bc8236af150021ea31ebc"
server_key_iv: "d1b1294264a8ee54c7156dab5793b931"
server_key_version: "encryptionkey@0"
I20260928 04:53:51.446026 5616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:53:51.446928 5630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:51.447165 5616 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:53:51.447297 5616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "879ea20f9b8042f9bc553dbfaf749286"
format_stamp: "Formatted at 2026-09-28 04:53:51 on dist-test-slave-m39l"
server_key: "13dc8ece277bc8236af150021ea31ebc"
server_key_iv: "d1b1294264a8ee54c7156dab5793b931"
server_key_version: "encryptionkey@0"
I20260928 04:53:51.447377 5616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:51.463366 5616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:51.463760 5616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:51.463938 5616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:51.468223 5616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43051
I20260928 04:53:51.468271 5682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43051 every 8 connection(s)
I20260928 04:53:51.468657 5616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:53:51.469638 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:51.472190 5683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286: Bootstrap starting.
I20260928 04:53:51.472870 5683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:51.473115 5683 log.cc:824] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:51.473766 5683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286: No bootstrap required, opened a new log
I20260928 04:53:51.475414 5683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879ea20f9b8042f9bc553dbfaf749286" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43051 } }
I20260928 04:53:51.475557 5683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:51.475581 5683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 879ea20f9b8042f9bc553dbfaf749286, State: Initialized, Role: FOLLOWER
I20260928 04:53:51.475693 5683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879ea20f9b8042f9bc553dbfaf749286" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43051 } }
I20260928 04:53:51.475775 5683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:53:51.475796 5683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:53:51.475822 5683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:51.478308 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 5616
I20260928 04:53:51.478389 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:53:51.478593 392 external_mini_cluster.cc:1428] Setting key 39f6a4e40d51e20940db7a2834893496
I20260928 04:53:51.481058 5683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879ea20f9b8042f9bc553dbfaf749286" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43051 } }
I20260928 04:53:51.481205 5683 leader_election.cc:302] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 879ea20f9b8042f9bc553dbfaf749286; no voters:
I20260928 04:53:51.481388 5683 leader_election.cc:288] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:53:51.481634 5688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:51.481730 5683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:51.481843 5688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286 [term 1 LEADER]: Becoming Leader. State: Replica: 879ea20f9b8042f9bc553dbfaf749286, State: Running, Role: LEADER
I20260928 04:53:51.481984 5688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879ea20f9b8042f9bc553dbfaf749286" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43051 } }
I20260928 04:53:51.482409 5688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "879ea20f9b8042f9bc553dbfaf749286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879ea20f9b8042f9bc553dbfaf749286" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43051 } } }
I20260928 04:53:51.482575 5688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:51.482755 5688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 879ea20f9b8042f9bc553dbfaf749286. Latest consensus state: current_term: 1 leader_uuid: "879ea20f9b8042f9bc553dbfaf749286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879ea20f9b8042f9bc553dbfaf749286" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43051 } } }
I20260928 04:53:51.482862 5688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:51.483027 5697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:53:51.484010 5697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:53:51.484566 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:51.479074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48174 (local address 127.0.98.62:43051)
0928 04:53:51.479314 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:51.479322 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:51.479350 (+ 28us) server_negotiation.cc:407] Connection header received
0928 04:53:51.479453 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:51.479459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:51.479515 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:51.479662 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:51.479915 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:51.480829 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:51.481923 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:51.482579 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:51.482695 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:51.483700 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:51.483703 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:51.483704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:53:51.483929 (+ 225us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:51.483975 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:53:51.483991 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:51.484065 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:51.484267 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 04:53:51.484352 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":89,"threads_started":1}
I20260928 04:53:51.485984 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:53:51.486258 5697 catalog_manager.cc:1347] Generated new cluster ID: 71b0548bee0e46808edd1af0f5a66f7c
I20260928 04:53:51.486332 5697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:53:51.496007 5697 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:53:51.496752 5697 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:53:51.515954 5697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 879ea20f9b8042f9bc553dbfaf749286: Generated new TSK 0
I20260928 04:53:51.516222 5697 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 04:53:51.541404 5713 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 04:53:51.541707 5713 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 04:53:51.542145 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:51.526355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48178 (local address 127.0.98.62:43051)
0928 04:53:51.526424 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:51.526427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:51.526606 (+ 179us) server_negotiation.cc:407] Connection header received
0928 04:53:51.526690 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:51.526691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:51.526715 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:51.526783 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0928 04:53:51.526939 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:51.527505 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:51.528741 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:51.529189 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:51.529282 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:51.529557 (+ 275us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 04:53:51.541916 (+ 12359us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 04:53:51.542087 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48178: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 04:53:51.542227 5687 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48178: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 04:53:51.542426 5712 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:53:51.526300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43051 (local address 127.0.0.1:48178)
0928 04:53:51.526521 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:51.526533 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:51.526607 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:51.526804 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:51.526805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:51.526807 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 04:53:51.526870 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:51.526872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:51.527599 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:51.527600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:51.528577 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:51.528581 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:51.529308 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:51.529314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:51.529403 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:51.529410 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 04:53:51.542054 (+ 12644us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 04:53:51.542227 (+ 173us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43051: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":131,"threads_started":1}
I20260928 04:53:51.542768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 5616
2026-09-28T04:53:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (247 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 04:53:51.581180 392 mini_oidc.cc:152] Starting JWKS server
I20260928 04:53:51.581892 392 webserver.cc:460] Webserver started at https://127.0.0.1:35919/ using document root <none> and password file <none>
I20260928 04:53:51.581965 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 04:53:51.582154 392 webserver.cc:460] Webserver started at http://127.0.0.1:44173/ using document root <none> and password file <none>
2026-09-28T04:53:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:53:51Z Disabled control of system clock
I20260928 04:53:51.591483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36453
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36453
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35919/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 04:53:51.674589 5719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:51.674825 5719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:51.674849 5719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:51.676153 5719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:53:51.676214 5719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:51.676227 5719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:51.676239 5719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:53:51.676254 5719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 04:53:51.676266 5719 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 04:53:51.676277 5719 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35919/jwks.json
I20260928 04:53:51.677827 5719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35919/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:51.678117 5719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:51.678434 5719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:51.683751 5727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:51.683751 5724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:51.684032 5719 server_base.cc:1034] running on GCE node
W20260928 04:53:51.683784 5725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:51.684370 5719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:51.684693 5719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:51.685868 5719 hybrid_clock.cc:648] HybridClock initialized: now 1790571231685840 us; error 39 us; skew 500 ppm
I20260928 04:53:51.687255 5719 webserver.cc:460] Webserver started at http://127.0.98.62:34949/ using document root <none> and password file <none>
I20260928 04:53:51.687507 5719 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:51.687582 5719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:51.687793 5719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:51.688915 5719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "425f269d064a426491a5d465247ff96a"
format_stamp: "Formatted at 2026-09-28 04:53:51 on dist-test-slave-m39l"
server_key: "76fdd4bb59a4e7452435404409414c33"
server_key_iv: "5c57ecc61cdc3957edd0398ec1c280dc"
server_key_version: "encryptionkey@0"
I20260928 04:53:51.689298 5719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "425f269d064a426491a5d465247ff96a"
format_stamp: "Formatted at 2026-09-28 04:53:51 on dist-test-slave-m39l"
server_key: "76fdd4bb59a4e7452435404409414c33"
server_key_iv: "5c57ecc61cdc3957edd0398ec1c280dc"
server_key_version: "encryptionkey@0"
I20260928 04:53:51.690852 5719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:53:51.691712 5733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:51.691923 5719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:51.692031 5719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "425f269d064a426491a5d465247ff96a"
format_stamp: "Formatted at 2026-09-28 04:53:51 on dist-test-slave-m39l"
server_key: "76fdd4bb59a4e7452435404409414c33"
server_key_iv: "5c57ecc61cdc3957edd0398ec1c280dc"
server_key_version: "encryptionkey@0"
I20260928 04:53:51.692108 5719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:51.713892 5719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:51.714257 5719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:51.714425 5719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:51.718418 5719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36453
I20260928 04:53:51.718526 5785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36453 every 8 connection(s)
I20260928 04:53:51.718873 5719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:53:51.719847 5786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:51.722183 5786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a: Bootstrap starting.
I20260928 04:53:51.722815 5786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:51.723050 5786 log.cc:824] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:51.723726 5786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a: No bootstrap required, opened a new log
I20260928 04:53:51.725250 5786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425f269d064a426491a5d465247ff96a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36453 } }
I20260928 04:53:51.725389 5786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:51.725410 5786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 425f269d064a426491a5d465247ff96a, State: Initialized, Role: FOLLOWER
I20260928 04:53:51.725515 5786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425f269d064a426491a5d465247ff96a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36453 } }
I20260928 04:53:51.725598 5786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:53:51.725631 5786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:53:51.725696 5786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:51.726125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 5719
I20260928 04:53:51.726203 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:53:51.726366 392 external_mini_cluster.cc:1428] Setting key 5cd7fe91738ecd6f0e1f6a6e236b6619
I20260928 04:53:51.731304 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:51.726903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.0.98.62:36453)
0928 04:53:51.727133 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:51.727147 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:51.727175 (+ 28us) server_negotiation.cc:407] Connection header received
0928 04:53:51.727268 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:51.727274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:51.727327 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:51.727438 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:51.727607 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:51.728757 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:51.729465 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:51.729904 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:51.729995 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:51.730603 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:51.730607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:51.730607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:53:51.730815 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:51.730851 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:53:51.730862 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:51.730913 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:51.730998 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 04:53:51.731108 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":118,"threads_started":1}
I20260928 04:53:51.732285 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:53:51.786003 5786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425f269d064a426491a5d465247ff96a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36453 } }
I20260928 04:53:51.786223 5786 leader_election.cc:302] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 425f269d064a426491a5d465247ff96a; no voters:
I20260928 04:53:51.786477 5786 leader_election.cc:288] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:53:51.786635 5791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:51.786824 5791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a [term 1 LEADER]: Becoming Leader. State: Replica: 425f269d064a426491a5d465247ff96a, State: Running, Role: LEADER
I20260928 04:53:51.786863 5786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:51.786926 5791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425f269d064a426491a5d465247ff96a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36453 } }
I20260928 04:53:51.787420 5792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "425f269d064a426491a5d465247ff96a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425f269d064a426491a5d465247ff96a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36453 } } }
I20260928 04:53:51.787509 5792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:51.787956 5793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 425f269d064a426491a5d465247ff96a. Latest consensus state: current_term: 1 leader_uuid: "425f269d064a426491a5d465247ff96a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425f269d064a426491a5d465247ff96a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36453 } } }
I20260928 04:53:51.788040 5793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:51.788314 5799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:53:51.788998 5799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:53:51.790925 5799 catalog_manager.cc:1347] Generated new cluster ID: f8e8812cdf674a029fc2371a43f99894
I20260928 04:53:51.790994 5799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:53:51.827152 5799 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:53:51.827751 5799 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:53:51.835150 5799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 425f269d064a426491a5d465247ff96a: Generated new TSK 0
I20260928 04:53:51.835398 5799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:53:51.852335 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:51.848257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49504 (local address 127.0.98.62:36453)
0928 04:53:51.848335 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:51.848338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:51.848498 (+ 160us) server_negotiation.cc:407] Connection header received
0928 04:53:51.848543 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:51.848544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:51.848564 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:51.848663 (+ 99us) server_negotiation.cc:226] Negotiated authn=JWT
0928 04:53:51.848821 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:51.849547 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:51.850384 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:51.850803 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:51.850884 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:51.851094 (+ 210us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 04:53:51.852090 (+ 996us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35919/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 04:53:51.852232 (+ 142us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49504: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35919/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 04:53:51.852406 5790 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49504: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35919/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 04:53:51.854409 5815 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:53:51.848209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36453 (local address 127.0.0.1:49504)
0928 04:53:51.848420 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:51.848430 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:51.848489 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:51.848685 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:51.848686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:51.848688 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 04:53:51.848749 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:51.848751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:51.849640 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:51.849640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:51.850293 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:51.850295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:51.850903 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:51.850913 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:51.850965 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:51.850971 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 04:53:51.852208 (+ 1237us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35919/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 04:53:51.854309 (+ 2101us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36453: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35919/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":1876,"thread_start_us":132,"threads_started":1}
I20260928 04:53:51.854925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 5719
E20260928 04:53:51.863785 5816 webserver.cc:533] Webserver: sslize failed:
E20260928 04:53:51.863878 5816 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T04:53:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (311 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T04:53:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:53:51Z Disabled control of system clock
I20260928 04:53:51.901093 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43853
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43853
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/testchainca.cert with env {}
W20260928 04:53:51.984592 5820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:51.984810 5820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:51.984834 5820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:51.986042 5820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:53:51.986088 5820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:51.986101 5820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:51.986114 5820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:53:51.986126 5820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 04:53:51.986188 5820 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 04:53:51.986204 5820 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 04:53:51.987736 5820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:51.988008 5820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:51.988245 5820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:51.993281 5826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:51.993275 5825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:51.993275 5828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:51.993750 5820 server_base.cc:1034] running on GCE node
I20260928 04:53:51.993993 5820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:51.994288 5820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:51.995458 5820 hybrid_clock.cc:648] HybridClock initialized: now 1790571231995421 us; error 56 us; skew 500 ppm
I20260928 04:53:51.996856 5820 webserver.cc:460] Webserver started at http://127.0.98.62:38183/ using document root <none> and password file <none>
I20260928 04:53:51.997120 5820 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:51.997192 5820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:51.997342 5820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:51.998448 5820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "b603359ab9974e99b6d8c9d4c3296539"
format_stamp: "Formatted at 2026-09-28 04:53:51 on dist-test-slave-m39l"
server_key: "dd6c2ea009b181efa9d37155fef1a616"
server_key_iv: "e250637933fca9deca6e4df4fd9a16ef"
server_key_version: "encryptionkey@0"
I20260928 04:53:51.998907 5820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "b603359ab9974e99b6d8c9d4c3296539"
format_stamp: "Formatted at 2026-09-28 04:53:51 on dist-test-slave-m39l"
server_key: "dd6c2ea009b181efa9d37155fef1a616"
server_key_iv: "e250637933fca9deca6e4df4fd9a16ef"
server_key_version: "encryptionkey@0"
I20260928 04:53:52.000510 5820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:53:52.001579 5834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:52.001783 5820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:52.001900 5820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "b603359ab9974e99b6d8c9d4c3296539"
format_stamp: "Formatted at 2026-09-28 04:53:51 on dist-test-slave-m39l"
server_key: "dd6c2ea009b181efa9d37155fef1a616"
server_key_iv: "e250637933fca9deca6e4df4fd9a16ef"
server_key_version: "encryptionkey@0"
I20260928 04:53:52.001991 5820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:52.014745 5820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:52.015044 5820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:52.015152 5820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:52.019716 5820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43853
I20260928 04:53:52.019749 5886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43853 every 8 connection(s)
I20260928 04:53:52.020145 5820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:53:52.021243 5887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:52.023864 5887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539: Bootstrap starting.
I20260928 04:53:52.024501 5887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:52.024813 5887 log.cc:824] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:52.025837 5887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539: No bootstrap required, opened a new log
I20260928 04:53:52.025888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 5820
I20260928 04:53:52.026003 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:53:52.026216 392 external_mini_cluster.cc:1428] Setting key f746048a239babc583f95b7fd4db8c3c
I20260928 04:53:52.028759 5887 raft_consensus.cc:348] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b603359ab9974e99b6d8c9d4c3296539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43853 } }
I20260928 04:53:52.029012 5887 raft_consensus.cc:374] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:52.029057 5887 raft_consensus.cc:729] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b603359ab9974e99b6d8c9d4c3296539, State: Initialized, Role: FOLLOWER
I20260928 04:53:52.029201 5887 consensus_queue.cc:260] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b603359ab9974e99b6d8c9d4c3296539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43853 } }
I20260928 04:53:52.029299 5887 raft_consensus.cc:388] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:53:52.029333 5887 raft_consensus.cc:482] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:53:52.029384 5887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:52.031121 5887 raft_consensus.cc:504] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b603359ab9974e99b6d8c9d4c3296539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43853 } }
I20260928 04:53:52.031241 5887 leader_election.cc:302] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b603359ab9974e99b6d8c9d4c3296539; no voters:
I20260928 04:53:52.031404 5887 leader_election.cc:288] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:53:52.031571 5892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:52.031701 5887 sys_catalog.cc:564] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:52.031821 5892 raft_consensus.cc:686] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539 [term 1 LEADER]: Becoming Leader. State: Replica: b603359ab9974e99b6d8c9d4c3296539, State: Running, Role: LEADER
I20260928 04:53:52.031991 5892 consensus_queue.cc:237] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b603359ab9974e99b6d8c9d4c3296539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43853 } }
I20260928 04:53:52.032431 5892 sys_catalog.cc:455] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b603359ab9974e99b6d8c9d4c3296539" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b603359ab9974e99b6d8c9d4c3296539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43853 } } }
I20260928 04:53:52.032560 5892 sys_catalog.cc:458] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:52.032732 5893 sys_catalog.cc:455] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b603359ab9974e99b6d8c9d4c3296539. Latest consensus state: current_term: 1 leader_uuid: "b603359ab9974e99b6d8c9d4c3296539" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b603359ab9974e99b6d8c9d4c3296539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43853 } } }
I20260928 04:53:52.032814 5893 sys_catalog.cc:458] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:52.033195 5902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:53:52.033775 5902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:53:52.035421 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:52.026877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57188 (local address 127.0.98.62:43853)
0928 04:53:52.027115 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:52.027126 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:52.027165 (+ 39us) server_negotiation.cc:407] Connection header received
0928 04:53:52.028710 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:52.028722 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:52.028835 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:52.029289 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:52.029413 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:52.031033 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:52.032652 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:52.033449 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:52.033584 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:52.034633 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:52.034638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:52.034638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:53:52.034877 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:52.034918 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:53:52.034934 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:52.034995 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:52.035129 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 04:53:52.035209 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
I20260928 04:53:52.036288 5902 catalog_manager.cc:1347] Generated new cluster ID: 3b7e15212b054c449de87dc05bf1f576
I20260928 04:53:52.036345 5902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:53:52.036332 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:53:52.046852 5902 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:53:52.047434 5902 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:53:52.053751 5902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b603359ab9974e99b6d8c9d4c3296539: Generated new TSK 0
I20260928 04:53:52.053984 5902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:53:52.496207 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:52.064942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.0.98.62:43853)
0928 04:53:52.064994 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:52.064997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:52.065098 (+ 101us) server_negotiation.cc:407] Connection header received
0928 04:53:52.065182 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:52.065183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:52.065194 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:52.065267 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0928 04:53:52.065416 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:52.065938 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:52.067116 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:52.067312 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:52.067368 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:52.067561 (+ 193us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 04:53:52.495933 (+428372us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 04:53:52.496143 (+ 210us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57204: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 04:53:52.496305 5891 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57204: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 04:53:52.496335 5916 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:53:52.064837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43853 (local address 127.0.0.1:57204)
0928 04:53:52.065019 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:52.065031 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:52.065104 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:52.065283 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:52.065284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:52.065285 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0928 04:53:52.065349 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:52.065351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:52.066023 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:52.066024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:52.067035 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:52.067038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:52.067393 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:52.067398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:52.067460 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:52.067465 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 04:53:52.496088 (+428623us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 04:53:52.496238 (+ 150us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43853: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":112,"threads_started":1}
I20260928 04:53:52.496858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 5820
2026-09-28T04:53:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (622 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (97 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:53:52 dist-test-slave-m39l krb5kdc[5928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:53:52 dist-test-slave-m39l krb5kdc[5928](info): set up 2 sockets
Sep 28 04:53:52 dist-test-slave-m39l krb5kdc[5928](info): commencing operation
krb5kdc: starting...
W20260928 04:53:53.762800 392 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:53:53 dist-test-slave-m39l krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:53:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:53:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:53:53.890702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38311
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:53.972499 5941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:53.972734 5941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:53.972764 5941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:53.974113 5941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:53:53.974161 5941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:53.974175 5941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:53.974189 5941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:53:53.974201 5941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:53:53.975947 5941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:53.976230 5941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:53.976481 5941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:53.981637 5947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:53.981644 5949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:53.981675 5946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:53.981928 5941 server_base.cc:1034] running on GCE node
I20260928 04:53:53.982173 5941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:53.982435 5941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:53.983585 5941 hybrid_clock.cc:648] HybridClock initialized: now 1790571233983548 us; error 50 us; skew 500 ppm
Sep 28 04:53:53 dist-test-slave-m39l krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:53.986145 5941 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:53:53.986737 5941 webserver.cc:460] Webserver started at http://127.0.98.62:42281/ using document root <none> and password file <none>
I20260928 04:53:53.986982 5941 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:53.987078 5941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:53.987288 5941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:53.988487 5941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "0e24bd8c7c1e41129ece7110557b6344"
format_stamp: "Formatted at 2026-09-28 04:53:53 on dist-test-slave-m39l"
server_key: "0f3596bfc56a806914c09ee0a49cd24e"
server_key_iv: "7c98fbcd5e536da8732ab9eebe5b9c57"
server_key_version: "encryptionkey@0"
I20260928 04:53:53.988874 5941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "0e24bd8c7c1e41129ece7110557b6344"
format_stamp: "Formatted at 2026-09-28 04:53:53 on dist-test-slave-m39l"
server_key: "0f3596bfc56a806914c09ee0a49cd24e"
server_key_iv: "7c98fbcd5e536da8732ab9eebe5b9c57"
server_key_version: "encryptionkey@0"
I20260928 04:53:53.990319 5941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:53:53.991139 5956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:53.991380 5941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:53.991497 5941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "0e24bd8c7c1e41129ece7110557b6344"
format_stamp: "Formatted at 2026-09-28 04:53:53 on dist-test-slave-m39l"
server_key: "0f3596bfc56a806914c09ee0a49cd24e"
server_key_iv: "7c98fbcd5e536da8732ab9eebe5b9c57"
server_key_version: "encryptionkey@0"
I20260928 04:53:53.991590 5941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:54.004151 5941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:54.006425 5941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:54.006662 5941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:54.011739 5941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38311
I20260928 04:53:54.011816 6008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38311 every 8 connection(s)
I20260928 04:53:54.012540 5941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:53:54.013190 6009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:54.015676 6009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344: Bootstrap starting.
I20260928 04:53:54.015872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 5941
I20260928 04:53:54.015971 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:53:54.016184 392 external_mini_cluster.cc:1428] Setting key 251fbc95ef40aa433eeab4ca8eb6f864
I20260928 04:53:54.016448 6009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:54.016821 6009 log.cc:824] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:54.017519 6009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344: No bootstrap required, opened a new log
I20260928 04:53:54.019861 6009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e24bd8c7c1e41129ece7110557b6344" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } }
I20260928 04:53:54.020001 6009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:54.020062 6009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e24bd8c7c1e41129ece7110557b6344, State: Initialized, Role: FOLLOWER
I20260928 04:53:54.020200 6009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e24bd8c7c1e41129ece7110557b6344" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } }
I20260928 04:53:54.020296 6009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:53:54.020341 6009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:53:54.020391 6009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:54.020938 6009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e24bd8c7c1e41129ece7110557b6344" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } }
I20260928 04:53:54.021054 6009 leader_election.cc:302] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e24bd8c7c1e41129ece7110557b6344; no voters:
I20260928 04:53:54.021215 6009 leader_election.cc:288] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:53:54.021379 6015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:54.021483 6009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:54.021552 6015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344 [term 1 LEADER]: Becoming Leader. State: Replica: 0e24bd8c7c1e41129ece7110557b6344, State: Running, Role: LEADER
I20260928 04:53:54.021667 6015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e24bd8c7c1e41129ece7110557b6344" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } }
I20260928 04:53:54.022040 6014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e24bd8c7c1e41129ece7110557b6344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e24bd8c7c1e41129ece7110557b6344" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } } }
I20260928 04:53:54.022128 6014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:54.022387 6016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e24bd8c7c1e41129ece7110557b6344. Latest consensus state: current_term: 1 leader_uuid: "0e24bd8c7c1e41129ece7110557b6344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e24bd8c7c1e41129ece7110557b6344" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } } }
I20260928 04:53:54.022724 6016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344 [sys.catalog]: This master's current role is: LEADER
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:54.023307 6021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:53:54.024010 6021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:53:54.027719 6021 catalog_manager.cc:1347] Generated new cluster ID: ac90552930f7447d933caa16bbe19d4f
I20260928 04:53:54.027801 6021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:53:54.029978 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:54.016694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.0.98.62:38311)
0928 04:53:54.016893 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:54.016903 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:54.016936 (+ 33us) server_negotiation.cc:407] Connection header received
0928 04:53:54.017507 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:54.017515 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:54.017573 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:54.017790 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:54.018434 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.019384 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.020183 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.020682 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.020799 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:54.023825 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:54.023846 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:54.023846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:54.023860 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:54.027898 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:54.028363 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:54.028371 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:54.028371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:54.028478 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:54.028672 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:54.028673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:54.028673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:54.028980 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:54.029138 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:54.029618 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 04:53:54.029761 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260928 04:53:54.037256 6021 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:53:54.038028 6021 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:53:54.044533 6021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e24bd8c7c1e41129ece7110557b6344: Generated new TSK 0
I20260928 04:53:54.044765 6021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:53:54.082238 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38311
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:54.165851 6035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:54.166076 6035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:54.166101 6035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:54.167304 6035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:54.167347 6035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:54.167374 6035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:53:54.168881 6035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:54.169143 6035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:54.169392 6035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:54.174769 6040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:54.174777 6043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:54.174777 6041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:54.175201 6035 server_base.cc:1034] running on GCE node
I20260928 04:53:54.175469 6035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:54.175771 6035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:54.176975 6035 hybrid_clock.cc:648] HybridClock initialized: now 1790571234176958 us; error 27 us; skew 500 ppm
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:54.180029 6035 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:53:54.180593 6035 webserver.cc:460] Webserver started at http://127.0.98.1:33201/ using document root <none> and password file <none>
I20260928 04:53:54.180846 6035 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:54.180919 6035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:54.181080 6035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:54.182230 6035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "f03657f8ae434367a57f3785d2c627cd"
format_stamp: "Formatted at 2026-09-28 04:53:54 on dist-test-slave-m39l"
server_key: "b46ef547d7f5e637f68942423a353dcf"
server_key_iv: "b0b78d0308869ccc53db21455bef131b"
server_key_version: "encryptionkey@0"
I20260928 04:53:54.182667 6035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f03657f8ae434367a57f3785d2c627cd"
format_stamp: "Formatted at 2026-09-28 04:53:54 on dist-test-slave-m39l"
server_key: "b46ef547d7f5e637f68942423a353dcf"
server_key_iv: "b0b78d0308869ccc53db21455bef131b"
server_key_version: "encryptionkey@0"
I20260928 04:53:54.184122 6035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:53:54.184903 6050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:54.185204 6035 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:53:54.185313 6035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "f03657f8ae434367a57f3785d2c627cd"
format_stamp: "Formatted at 2026-09-28 04:53:54 on dist-test-slave-m39l"
server_key: "b46ef547d7f5e637f68942423a353dcf"
server_key_iv: "b0b78d0308869ccc53db21455bef131b"
server_key_version: "encryptionkey@0"
I20260928 04:53:54.185391 6035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:54.207595 6035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:54.209889 6035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:54.210072 6035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:54.210323 6035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:54.210711 6035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:54.210773 6035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:54.210806 6035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:54.210829 6035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:54.216859 6035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42993
I20260928 04:53:54.217024 6163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42993 every 8 connection(s)
I20260928 04:53:54.217284 6035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:53:54.217741 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 6035
I20260928 04:53:54.217864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:53:54.218066 392 external_mini_cluster.cc:1428] Setting key 9e44df6dfddfcc1ddca36868101f17e5
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 04:53:54.230060 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:54.218727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57295 (local address 127.0.98.62:38311)
0928 04:53:54.218775 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:54.218777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:54.219044 (+ 267us) server_negotiation.cc:407] Connection header received
0928 04:53:54.220759 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:54.220761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:54.220776 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:54.220850 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:54.221738 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.222614 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.224010 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.224379 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.224464 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:54.226955 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:54.226956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:54.226957 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:54.226966 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:54.228345 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:54.228924 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:54.228926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:54.228926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:54.228997 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:54.229261 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:54.229262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:54.229262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:54.229420 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:54.229490 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:54.229876 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 04:53:54.229974 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:53:54.230334 6166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:54.218718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:57295)
0928 04:53:54.218905 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:54.218957 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:54.220515 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:54.220938 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:54.220942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:54.221314 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:54.221535 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:54.221541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.222808 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.222808 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:54.223913 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:54.223916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.224474 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.224475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:54.224584 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:54.225250 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:54.225260 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:54.226834 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:54.228424 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:54.228430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:54.228432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:54.228847 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:54.229063 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:54.229064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:54.229064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:54.229178 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:54.229511 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:54.229518 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:54.229673 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 04:53:54.229835 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 04:53:54.230019 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260928 04:53:54.230950 6164 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38311
I20260928 04:53:54.231066 6164 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:54.231307 6164 heartbeater.cc:511] Master 127.0.98.62:38311 requested a full tablet report, sending...
I20260928 04:53:54.231993 5973 ts_manager.cc:194] Registered new tserver with Master: f03657f8ae434367a57f3785d2c627cd (127.0.98.1:42993)
I20260928 04:53:54.232681 5973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:53:54.245920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38311
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:54.329950 6169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:54.330188 6169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:54.330215 6169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:54.331571 6169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:54.331631 6169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:54.331660 6169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:53:54.333280 6169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:54.333549 6169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:54.333832 6169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:54.339174 6175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:54.339267 6169 server_base.cc:1034] running on GCE node
W20260928 04:53:54.339275 6177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:54.339184 6174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:54.339643 6169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:54.339922 6169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:54.341090 6169 hybrid_clock.cc:648] HybridClock initialized: now 1790571234341070 us; error 46 us; skew 500 ppm
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:54.343920 6169 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:53:54.344385 6169 webserver.cc:460] Webserver started at http://127.0.98.2:43879/ using document root <none> and password file <none>
I20260928 04:53:54.344636 6169 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:54.344707 6169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:54.344864 6169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:54.345978 6169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "acd3df894f7247cd888ba9bf258367ec"
format_stamp: "Formatted at 2026-09-28 04:53:54 on dist-test-slave-m39l"
server_key: "a9fc8d3dcb469fe4d369092f52d6494e"
server_key_iv: "7b6b58e94acab26e1d7fa1fca2d94016"
server_key_version: "encryptionkey@0"
I20260928 04:53:54.346359 6169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "acd3df894f7247cd888ba9bf258367ec"
format_stamp: "Formatted at 2026-09-28 04:53:54 on dist-test-slave-m39l"
server_key: "a9fc8d3dcb469fe4d369092f52d6494e"
server_key_iv: "7b6b58e94acab26e1d7fa1fca2d94016"
server_key_version: "encryptionkey@0"
I20260928 04:53:54.347863 6169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:53:54.348718 6184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:54.349074 6169 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:54.349180 6169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "acd3df894f7247cd888ba9bf258367ec"
format_stamp: "Formatted at 2026-09-28 04:53:54 on dist-test-slave-m39l"
server_key: "a9fc8d3dcb469fe4d369092f52d6494e"
server_key_iv: "7b6b58e94acab26e1d7fa1fca2d94016"
server_key_version: "encryptionkey@0"
I20260928 04:53:54.349330 6169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:54.371454 6169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:54.373759 6169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:54.373924 6169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:54.374163 6169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:54.374539 6169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:54.374596 6169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:54.374639 6169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:54.374693 6169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:54.380914 6169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46619
I20260928 04:53:54.380965 6297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46619 every 8 connection(s)
I20260928 04:53:54.381320 6169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:54.391036 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 6169
I20260928 04:53:54.391147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:53:54.391369 392 external_mini_cluster.cc:1428] Setting key 83d6a717e16cb5cef943230578fc6364
I20260928 04:53:54.392436 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:54.382195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39781 (local address 127.0.98.62:38311)
0928 04:53:54.382274 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:54.382276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:54.382546 (+ 270us) server_negotiation.cc:407] Connection header received
0928 04:53:54.383486 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:54.383488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:54.383499 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:54.383629 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:54.384295 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.384844 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.385908 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.386129 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.386199 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:54.388728 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:54.388730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:54.388730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:54.388739 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:54.390617 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:54.391188 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:54.391189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:54.391190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:54.391259 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:54.391552 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:54.391554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:54.391554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:54.391725 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:54.391810 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:54.392178 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 04:53:54.392308 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 04:53:54.392565 6300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:54.382172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:39781)
0928 04:53:54.382323 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:54.382367 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:54.383365 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:54.383688 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:54.383692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:54.383967 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:54.384171 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:54.384175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.384946 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.384947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:54.385801 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:54.385803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.386237 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.386238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:54.386358 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:54.387003 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:54.387013 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:54.388605 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:54.390702 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:54.390713 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:54.390716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:54.391067 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:54.391317 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:54.391318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:54.391318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:54.391479 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:54.391801 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:54.391809 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:54.392007 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 04:53:54.392145 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 04:53:54.392330 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260928 04:53:54.393105 6298 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38311
I20260928 04:53:54.393234 6298 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:54.393450 6298 heartbeater.cc:511] Master 127.0.98.62:38311 requested a full tablet report, sending...
I20260928 04:53:54.393949 5973 ts_manager.cc:194] Registered new tserver with Master: acd3df894f7247cd888ba9bf258367ec (127.0.98.2:46619)
I20260928 04:53:54.394445 5973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39781
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:53:54.418840 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38311
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:54.501204 6303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:54.501436 6303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:54.501462 6303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:54.502686 6303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:54.502736 6303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:54.502763 6303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:53:54.504226 6303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:54.504420 6303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:54.504601 6303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:54.510092 6309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:54.510134 6308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:54.510236 6303 server_base.cc:1034] running on GCE node
W20260928 04:53:54.510134 6311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:54.510664 6303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:54.511004 6303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:54.512208 6303 hybrid_clock.cc:648] HybridClock initialized: now 1790571234512196 us; error 53 us; skew 500 ppm
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:54.515463 6303 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:53:54.515931 6303 webserver.cc:460] Webserver started at http://127.0.98.3:46675/ using document root <none> and password file <none>
I20260928 04:53:54.516140 6303 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:54.516187 6303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:54.516288 6303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:54.517390 6303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "34ba21ddc2c14656b8f4460ea1864be7"
format_stamp: "Formatted at 2026-09-28 04:53:54 on dist-test-slave-m39l"
server_key: "a0e67adcbe6d25518a0c76532a2da67b"
server_key_iv: "8296b626bf18074e97caa545268819b9"
server_key_version: "encryptionkey@0"
I20260928 04:53:54.517746 6303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "34ba21ddc2c14656b8f4460ea1864be7"
format_stamp: "Formatted at 2026-09-28 04:53:54 on dist-test-slave-m39l"
server_key: "a0e67adcbe6d25518a0c76532a2da67b"
server_key_iv: "8296b626bf18074e97caa545268819b9"
server_key_version: "encryptionkey@0"
I20260928 04:53:54.519311 6303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:53:54.520135 6318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:54.520335 6303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:54.520421 6303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "34ba21ddc2c14656b8f4460ea1864be7"
format_stamp: "Formatted at 2026-09-28 04:53:54 on dist-test-slave-m39l"
server_key: "a0e67adcbe6d25518a0c76532a2da67b"
server_key_iv: "8296b626bf18074e97caa545268819b9"
server_key_version: "encryptionkey@0"
I20260928 04:53:54.520488 6303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:54.535681 6303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:54.537807 6303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:54.537936 6303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:54.538156 6303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:54.538558 6303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:54.538631 6303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:54.538666 6303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:54.538681 6303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:54.545413 6303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33341
I20260928 04:53:54.545452 6431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33341 every 8 connection(s)
I20260928 04:53:54.545814 6303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:53:54.554201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 6303
I20260928 04:53:54.554284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:53:54.554477 392 external_mini_cluster.cc:1428] Setting key 8acc50f694470f7ba0265c7900078c51
I20260928 04:53:54.557343 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:54.546694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54365 (local address 127.0.98.62:38311)
0928 04:53:54.546776 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:54.546779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:54.547058 (+ 279us) server_negotiation.cc:407] Connection header received
0928 04:53:54.548171 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:54.548172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:54.548184 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:54.548254 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:54.548953 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.549538 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.550850 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.551100 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.551176 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:54.553842 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:54.553844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:54.553844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:54.553852 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:54.555625 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:54.556117 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:54.556118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:54.556119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:54.556178 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:54.556440 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:54.556441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:54.556441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:54.556627 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:54.556697 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:54.557069 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 04:53:54.557144 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 04:53:54.557389 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:54.546690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:54365)
0928 04:53:54.546909 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:54.546951 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:54.548058 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:54.548315 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:54.548319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:54.548611 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:54.548833 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:54.548838 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.549655 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.549656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:54.550741 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:54.550745 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.551213 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.551214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:54.551327 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:54.551930 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:54.551939 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:54.553740 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:54.555755 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:54.555763 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:54.555765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:54.556023 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:54.556234 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:54.556234 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:54.556235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:54.556342 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:54.556735 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:54.556742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:54.556893 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 04:53:54.556999 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 04:53:54.557184 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
I20260928 04:53:54.558096 6432 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38311
I20260928 04:53:54.558228 6432 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:54.558468 6432 heartbeater.cc:511] Master 127.0.98.62:38311 requested a full tablet report, sending...
I20260928 04:53:54.559012 5973 ts_manager.cc:194] Registered new tserver with Master: 34ba21ddc2c14656b8f4460ea1864be7 (127.0.98.3:33341)
I20260928 04:53:54.559396 5973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54365
I20260928 04:53:54.566470 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/test-admin.keytab.
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:54.584376 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 04:53:54.586133 6439 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:53:54.585341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.0.98.1:42993)
0928 04:53:54.585515 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:54.585528 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:54.585540 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:53:54.585568 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:54.585570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:54.585584 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:54.585656 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:54.586019 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57574: BlockingRecv error: recv got EOF from 127.0.0.1:57574 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
W20260928 04:53:54.586174 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:54.587199 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:54.588204 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:53:54.597173 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:54.588742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.0.98.1:42993)
0928 04:53:54.588786 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:54.588788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:54.588927 (+ 139us) server_negotiation.cc:407] Connection header received
0928 04:53:54.589008 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:54.589009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:54.589029 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:54.589097 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:54.589507 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.590244 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.591077 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.591409 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.591503 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:54.593848 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:54.593857 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:54.593857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:54.593864 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:54.595920 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:54.596319 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:54.596322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:54.596322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:54.596383 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:54.596580 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:54.596581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:54.596581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:54.596760 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:54.596878 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:54.597016 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 04:53:54.597087 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 04:53:54.599437 6439 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:53:54.598617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.0.98.1:42993)
0928 04:53:54.598707 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:54.598709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:54.598837 (+ 128us) server_negotiation.cc:407] Connection header received
0928 04:53:54.598916 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:54.598917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:54.598931 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:54.599010 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:54.599380 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57584: BlockingRecv error: recv got EOF from 127.0.0.1:57584 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 04:53:54.599548 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:54.600679 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:54.601660 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:53:54.609565 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:54.602364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57588 (local address 127.0.98.1:42993)
0928 04:53:54.602437 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:54.602439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:54.602462 (+ 23us) server_negotiation.cc:407] Connection header received
0928 04:53:54.602582 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:54.602583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:54.602593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:54.602637 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:54.603046 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.603541 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.604280 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.604474 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.604543 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:54.606765 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:54.606766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:54.606766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:54.606773 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:54.608328 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:54.608777 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:54.608778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:54.608778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:54.608832 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:54.609017 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:54.609017 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:54.609017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:54.609119 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:54.609200 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:54.609324 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 04:53:54.609392 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 04:53:54.611514 6439 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:53:54.610920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57590 (local address 127.0.98.1:42993)
0928 04:53:54.610960 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:54.610962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:54.611018 (+ 56us) server_negotiation.cc:407] Connection header received
0928 04:53:54.611095 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:54.611096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:54.611104 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:54.611144 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:54.611446 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57590: BlockingRecv error: recv got EOF from 127.0.0.1:57590 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 04:53:54.611711 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790571208412331-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:54.612821 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:54.613742 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:53:54.621448 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:54.614329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57598 (local address 127.0.98.1:42993)
0928 04:53:54.614390 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:54.614391 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:54.614423 (+ 32us) server_negotiation.cc:407] Connection header received
0928 04:53:54.614601 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:54.614602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:54.614612 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:54.614669 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:54.615080 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.615611 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.616355 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:54.616550 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:54.616604 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:54.618831 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:54.618832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:54.618833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:54.618840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:54.620185 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:54.620515 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:54.620515 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:54.620516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:54.620624 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:54.620844 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:54.620845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:54.620845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:54.620951 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:54.621104 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:54.621221 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 04:53:54.621292 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:53:54.622575 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 6035
I20260928 04:53:54.627074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 6169
I20260928 04:53:54.631600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 6303
I20260928 04:53:54.636608 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 5941
2026-09-28T04:53:54Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1954 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[6452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[6452](info): set up 2 sockets
Sep 28 04:53:54 dist-test-slave-m39l krb5kdc[6452](info): commencing operation
krb5kdc: starting...
W20260928 04:53:55.682940 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:53:55 dist-test-slave-m39l krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:53:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:53:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 04:53:55.810245 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44343
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:55.892066 6465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:55.892364 6465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:55.892403 6465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:55.893648 6465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:53:55.893743 6465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:55.893769 6465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:55.893800 6465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:53:55.893848 6465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:53:55.895532 6465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:55.895845 6465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:55.896090 6465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:55.901526 6473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:55.901479 6471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:55.901472 6470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:55.901652 6465 server_base.cc:1034] running on GCE node
I20260928 04:53:55.901996 6465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:55.902287 6465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:55.903466 6465 hybrid_clock.cc:648] HybridClock initialized: now 1790571235903439 us; error 64 us; skew 500 ppm
Sep 28 04:53:55 dist-test-slave-m39l krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571235, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:55.906111 6465 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 04:53:55.906708 6465 webserver.cc:460] Webserver started at http://127.0.98.62:37227/ using document root <none> and password file <none>
I20260928 04:53:55.906940 6465 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:55.907011 6465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:55.907168 6465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:55.908296 6465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "55fc8a9c3dd54c1eaf1dece4e8993857"
format_stamp: "Formatted at 2026-09-28 04:53:55 on dist-test-slave-m39l"
server_key: "29377c9218b3bf4764fc7d1d5f84565d"
server_key_iv: "68703ee84ed7715157e843d5ec5d1efa"
server_key_version: "encryptionkey@0"
I20260928 04:53:55.908672 6465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "55fc8a9c3dd54c1eaf1dece4e8993857"
format_stamp: "Formatted at 2026-09-28 04:53:55 on dist-test-slave-m39l"
server_key: "29377c9218b3bf4764fc7d1d5f84565d"
server_key_iv: "68703ee84ed7715157e843d5ec5d1efa"
server_key_version: "encryptionkey@0"
I20260928 04:53:55.910095 6465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:53:55.911062 6480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:55.911396 6465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:53:55.911523 6465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "55fc8a9c3dd54c1eaf1dece4e8993857"
format_stamp: "Formatted at 2026-09-28 04:53:55 on dist-test-slave-m39l"
server_key: "29377c9218b3bf4764fc7d1d5f84565d"
server_key_iv: "68703ee84ed7715157e843d5ec5d1efa"
server_key_version: "encryptionkey@0"
I20260928 04:53:55.911633 6465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:55.924423 6465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:55.927362 6465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:55.927547 6465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:55.932437 6465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44343
I20260928 04:53:55.932492 6532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44343 every 8 connection(s)
I20260928 04:53:55.932947 6465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:53:55.934141 6533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:55.935350 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 6465
I20260928 04:53:55.935467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:53:55.935662 392 external_mini_cluster.cc:1428] Setting key 031d56b83299956d4ed6573775ae7c77
I20260928 04:53:55.937883 6533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857: Bootstrap starting.
I20260928 04:53:55.938773 6533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:55.939107 6533 log.cc:824] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:55.939909 6533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857: No bootstrap required, opened a new log
I20260928 04:53:55.942214 6533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } }
I20260928 04:53:55.942384 6533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:55.942407 6533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fc8a9c3dd54c1eaf1dece4e8993857, State: Initialized, Role: FOLLOWER
I20260928 04:53:55.942602 6533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } }
I20260928 04:53:55.942723 6533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:53:55.942776 6533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:53:55.942834 6533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 04:53:55 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:53:55.943524 6533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } }
I20260928 04:53:55.943651 6533 leader_election.cc:302] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55fc8a9c3dd54c1eaf1dece4e8993857; no voters:
I20260928 04:53:55.943871 6533 leader_election.cc:288] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:53:55.943953 6538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:55.944164 6533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:55.945379 6538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [term 1 LEADER]: Becoming Leader. State: Replica: 55fc8a9c3dd54c1eaf1dece4e8993857, State: Running, Role: LEADER
I20260928 04:53:55.945623 6538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } }
I20260928 04:53:55.946327 6539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } }
W20260928 04:53:55.947055 6552 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:53:55.947113 6552 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:53:55.946439 6539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:55.946585 6540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55fc8a9c3dd54c1eaf1dece4e8993857. Latest consensus state: current_term: 1 leader_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } }
I20260928 04:53:55.947664 6540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:55.947708 6556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:53:55.947914 6556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:53:55.949126 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:55.936263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.0.98.62:44343)
0928 04:53:55.936451 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:55.936461 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:55.936493 (+ 32us) server_negotiation.cc:407] Connection header received
0928 04:53:55.937037 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:55.937047 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:55.937120 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:55.937292 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:55.937923 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:55.939496 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:55.940393 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:55.941020 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:55.941143 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:55.943768 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:55.943773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:55.943774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:55.943789 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:55.947386 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:55.947864 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:55.947869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:55.947870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:55.947991 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:55.948208 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:55.948209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:55.948210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:55.948468 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:55.948627 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:55.948750 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 04:53:55.948897 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260928 04:53:55.949970 6556 catalog_manager.cc:1347] Generated new cluster ID: 7c486fd6474b4aacbbe0b93916d5be95
I20260928 04:53:55.950042 6556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:53:55.965899 6556 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:53:55.966542 6556 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:53:55.975589 6556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857: Generated new TSK 0
I20260928 04:53:55.975915 6556 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 04:53:56.011883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44343
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:56.103217 6560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:56.103546 6560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:56.103590 6560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:56.104943 6560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:56.105006 6560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:56.105037 6560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:53:56.106779 6560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:56.107091 6560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:56.107354 6560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:56.112926 6566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:56.112936 6565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:56.113097 6560 server_base.cc:1034] running on GCE node
W20260928 04:53:56.112948 6568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:56.113395 6560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:56.113692 6560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:56.114874 6560 hybrid_clock.cc:648] HybridClock initialized: now 1790571236114915 us; error 101 us; skew 500 ppm
Sep 28 04:53:56 dist-test-slave-m39l krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571236, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:56.117877 6560 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 04:53:56.118398 6560 webserver.cc:460] Webserver started at http://127.0.98.1:36849/ using document root <none> and password file <none>
I20260928 04:53:56.118719 6560 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:56.118798 6560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:56.118978 6560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:56.120193 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "db43418eb20a4d5ca92826ab4c6db24e"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "c7b3b275aa0f201d8f141c5f2d33c169"
server_key_iv: "0e8293ff533eb45e00c93aef385861f9"
server_key_version: "encryptionkey@0"
I20260928 04:53:56.120617 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "db43418eb20a4d5ca92826ab4c6db24e"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "c7b3b275aa0f201d8f141c5f2d33c169"
server_key_iv: "0e8293ff533eb45e00c93aef385861f9"
server_key_version: "encryptionkey@0"
I20260928 04:53:56.122143 6560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:53:56.123145 6575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:56.123394 6560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:53:56.123517 6560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "db43418eb20a4d5ca92826ab4c6db24e"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "c7b3b275aa0f201d8f141c5f2d33c169"
server_key_iv: "0e8293ff533eb45e00c93aef385861f9"
server_key_version: "encryptionkey@0"
I20260928 04:53:56.123615 6560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:56.144032 6560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:56.146921 6560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:56.147125 6560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:56.147401 6560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:56.147773 6560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:56.147841 6560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:56.147884 6560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:56.147918 6560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:56.154230 6560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40899
I20260928 04:53:56.154237 6688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40899 every 8 connection(s)
I20260928 04:53:56.154699 6560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:53:56.157881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 6560
I20260928 04:53:56.157963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:53:56.158207 392 external_mini_cluster.cc:1428] Setting key ed99985f80250a37a53e36750719eb43
Sep 28 04:53:56 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571236, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:53:56.166225 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.155644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38531 (local address 127.0.98.62:44343)
0928 04:53:56.155727 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:56.155729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:56.155892 (+ 163us) server_negotiation.cc:407] Connection header received
0928 04:53:56.156385 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:56.156386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:56.156396 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:56.156462 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:56.157536 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.158158 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.159764 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.159983 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.160046 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.162832 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:56.162834 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:56.162834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:56.162843 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:56.164548 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.165085 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.165087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.165088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.165166 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.165468 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.165469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.165469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.165603 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:56.165692 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:56.166002 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 04:53:56.166106 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 04:53:56.166430 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.155600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:38531)
0928 04:53:56.155760 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:56.155803 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:56.156297 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:56.156624 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:56.156628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:56.157239 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:56.157445 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:56.157451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.158287 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.158288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:56.159667 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:56.159671 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.160082 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.160083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:56.160235 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.160872 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:56.160882 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:56.162712 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:56.164666 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:56.164675 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:56.164685 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:56.164940 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:56.165254 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:56.165255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:56.165256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:56.165385 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:56.165686 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:56.165692 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:56.165855 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 04:53:56.165981 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 04:53:56.166162 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260928 04:53:56.167191 6689 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44343
I20260928 04:53:56.167326 6689 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:56.167562 6689 heartbeater.cc:511] Master 127.0.98.62:44343 requested a full tablet report, sending...
I20260928 04:53:56.168105 6497 ts_manager.cc:194] Registered new tserver with Master: db43418eb20a4d5ca92826ab4c6db24e (127.0.98.1:40899)
I20260928 04:53:56.168769 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 04:53:56.186237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44343
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:56.271142 6694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:56.271392 6694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:56.271420 6694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:56.272768 6694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:56.272830 6694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:56.272859 6694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:53:56.274482 6694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:56.274813 6694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:56.275075 6694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:56.280452 6700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:56.280565 6694 server_base.cc:1034] running on GCE node
W20260928 04:53:56.280447 6699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:56.280457 6702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:56.280884 6694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:56.281092 6694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:56.282245 6694 hybrid_clock.cc:648] HybridClock initialized: now 1790571236282234 us; error 25 us; skew 500 ppm
Sep 28 04:53:56 dist-test-slave-m39l krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571236, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:56.284801 6694 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 04:53:56.285271 6694 webserver.cc:460] Webserver started at http://127.0.98.2:34307/ using document root <none> and password file <none>
I20260928 04:53:56.285503 6694 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:56.285573 6694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:56.285727 6694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:56.286947 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "62ddf5590453482a9ada0d96856c81da"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "625fa203024fd4a7d9036c4ee19a26df"
server_key_iv: "01182343f6f27fbb989c030f506730e6"
server_key_version: "encryptionkey@0"
I20260928 04:53:56.287317 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "62ddf5590453482a9ada0d96856c81da"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "625fa203024fd4a7d9036c4ee19a26df"
server_key_iv: "01182343f6f27fbb989c030f506730e6"
server_key_version: "encryptionkey@0"
I20260928 04:53:56.288676 6694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:53:56.289388 6709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:56.289553 6694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:53:56.289654 6694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "62ddf5590453482a9ada0d96856c81da"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "625fa203024fd4a7d9036c4ee19a26df"
server_key_iv: "01182343f6f27fbb989c030f506730e6"
server_key_version: "encryptionkey@0"
I20260928 04:53:56.289739 6694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:56.315925 6694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:56.318346 6694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:56.318594 6694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:56.318874 6694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:56.319208 6694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:56.319264 6694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:56.319307 6694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:56.319335 6694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:56.325299 6694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41419
I20260928 04:53:56.325356 6822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41419 every 8 connection(s)
I20260928 04:53:56.325681 6694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:53:56.331815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 6694
I20260928 04:53:56.331911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:53:56.332172 392 external_mini_cluster.cc:1428] Setting key 487588292865fe8df3294664cbb00cf5
Sep 28 04:53:56 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571236, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:53:56.337040 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.326434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49305 (local address 127.0.98.62:44343)
0928 04:53:56.326559 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:56.326562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:56.326760 (+ 198us) server_negotiation.cc:407] Connection header received
0928 04:53:56.327390 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:56.327391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:56.327405 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:56.327464 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:56.328479 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.328993 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.330027 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.330246 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.330336 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.333355 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:56.333356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:56.333357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:56.333366 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:56.335230 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.335871 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.335873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.335873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.335939 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.336305 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.336306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.336306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.336438 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:56.336498 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:56.336870 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 04:53:56.336969 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:53:56.337353 6825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.326435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:49305)
0928 04:53:56.326636 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:56.326674 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:56.327282 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:56.327546 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:56.327549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:56.328175 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:56.328380 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:56.328386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.329096 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.329098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:56.329932 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:56.329934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.330350 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.330351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:56.330476 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.331096 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:56.331105 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:56.333248 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:56.335362 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:56.335374 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:56.335377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:56.335762 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:56.336018 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:56.336019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:56.336019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:56.336225 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:56.336511 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:56.336518 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:56.336720 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 04:53:56.336846 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 04:53:56.337055 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":118,"threads_started":1}
I20260928 04:53:56.337965 6823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44343
I20260928 04:53:56.338109 6823 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:56.338331 6823 heartbeater.cc:511] Master 127.0.98.62:44343 requested a full tablet report, sending...
I20260928 04:53:56.338812 6497 ts_manager.cc:194] Registered new tserver with Master: 62ddf5590453482a9ada0d96856c81da (127.0.98.2:41419)
I20260928 04:53:56.339401 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49305
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 04:53:56.361781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44343
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:56.448053 6828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:56.448284 6828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:56.448310 6828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:56.449710 6828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:56.449762 6828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:56.449791 6828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:53:56.451630 6828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:56.451850 6828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:56.452062 6828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:56.457374 6834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:56.457368 6836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:56.457368 6833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:56.457551 6828 server_base.cc:1034] running on GCE node
I20260928 04:53:56.457885 6828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:56.458191 6828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:56.459414 6828 hybrid_clock.cc:648] HybridClock initialized: now 1790571236459381 us; error 41 us; skew 500 ppm
Sep 28 04:53:56 dist-test-slave-m39l krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571236, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:56.462797 6828 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 04:53:56.463393 6828 webserver.cc:460] Webserver started at http://127.0.98.3:32861/ using document root <none> and password file <none>
I20260928 04:53:56.463646 6828 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:56.463728 6828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:56.463886 6828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:56.465011 6828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "ea50a1c39d174993ae777a27876c8403"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "25dd70080156fc373f4b02c186b7933b"
server_key_iv: "a1348cb7ce916b12ab69a1e2532008d5"
server_key_version: "encryptionkey@0"
I20260928 04:53:56.465413 6828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ea50a1c39d174993ae777a27876c8403"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "25dd70080156fc373f4b02c186b7933b"
server_key_iv: "a1348cb7ce916b12ab69a1e2532008d5"
server_key_version: "encryptionkey@0"
I20260928 04:53:56.467140 6828 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 04:53:56.468112 6843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:56.468314 6828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:56.468436 6828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "ea50a1c39d174993ae777a27876c8403"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "25dd70080156fc373f4b02c186b7933b"
server_key_iv: "a1348cb7ce916b12ab69a1e2532008d5"
server_key_version: "encryptionkey@0"
I20260928 04:53:56.468598 6828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:56.477283 6828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:56.479611 6828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:56.479825 6828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:56.480082 6828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:56.480403 6828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:56.480459 6828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:56.480504 6828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:56.480528 6828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:56.486341 6828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36755
I20260928 04:53:56.486399 6956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36755 every 8 connection(s)
I20260928 04:53:56.486838 6828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:53:56.487459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 6828
I20260928 04:53:56.487525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:53:56.487759 392 external_mini_cluster.cc:1428] Setting key 0ff75a222b7cd61d156128ebac9db911
Sep 28 04:53:56 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571236, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:53:56.497433 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.487958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44903 (local address 127.0.98.62:44343)
0928 04:53:56.488029 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:56.488031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:56.488041 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:53:56.488470 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:56.488472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:56.488484 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:56.488531 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:56.489504 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.490019 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.491112 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.491334 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.491425 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.494026 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:56.494028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:56.494029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:56.494039 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:56.495688 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.496203 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.496205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.496206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.496274 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.496541 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.496541 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.496542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.496775 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:56.496893 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:56.497240 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 04:53:56.497332 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 04:53:56.497821 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.487652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:44903)
0928 04:53:56.487809 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:56.487853 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:56.488367 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:56.488605 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:56.488610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:56.489194 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:56.489396 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:56.489401 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.490123 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.490124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:56.491017 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:56.491020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.491434 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.491436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:56.491563 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.492097 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:56.492106 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:56.493885 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:56.495817 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:56.495829 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:56.495847 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:56.496111 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:56.496344 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:56.496345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:56.496345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:56.496472 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:56.496882 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:56.496890 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:56.497081 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 04:53:56.497204 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 04:53:56.497392 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260928 04:53:56.498471 6957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44343
I20260928 04:53:56.498648 6957 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:56.498863 6957 heartbeater.cc:511] Master 127.0.98.62:44343 requested a full tablet report, sending...
I20260928 04:53:56.499323 6496 ts_manager.cc:194] Registered new tserver with Master: ea50a1c39d174993ae777a27876c8403 (127.0.98.3:36755)
I20260928 04:53:56.499783 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44903
I20260928 04:53:56.510524 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260928 04:53:56.542171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43647
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44343,127.0.98.61:43647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:56.625430 6962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:56.625666 6962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:56.625697 6962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:56.626956 6962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:53:56.627007 6962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:56.627022 6962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:56.627035 6962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:53:56.627048 6962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:53:56.628664 6962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44343,127.0.98.61:43647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
W20260928 04:53:56.628784 6962 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:44343,127.0.98.61:43647'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 04:53:56.628904 6962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:56.629091 6962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:56.634310 6967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:56.634286 6968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:56.634286 6970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:56.634545 6962 server_base.cc:1034] running on GCE node
I20260928 04:53:56.634949 6962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:56.635299 6962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:56.636461 6962 hybrid_clock.cc:648] HybridClock initialized: now 1790571236636442 us; error 36 us; skew 500 ppm
Sep 28 04:53:56 dist-test-slave-m39l krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571236, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:56.639175 6962 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 04:53:56.639695 6962 webserver.cc:460] Webserver started at http://127.0.98.61:39359/ using document root <none> and password file <none>
I20260928 04:53:56.639948 6962 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:56.640022 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:56.640173 6962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:56.641263 6962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/data/instance:
uuid: "065816551ea64988a938a5e2e163db38"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "42f3ab0a2e915996767a39b12e92ce58"
server_key_iv: "c26f36c77e594d02fe7baec90d99ccc2"
server_key_version: "encryptionkey@0"
I20260928 04:53:56.641633 6962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/wal/instance:
uuid: "065816551ea64988a938a5e2e163db38"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "42f3ab0a2e915996767a39b12e92ce58"
server_key_iv: "c26f36c77e594d02fe7baec90d99ccc2"
server_key_version: "encryptionkey@0"
I20260928 04:53:56.643172 6962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:53:56.643989 6977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:56.644173 6962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:53:56.644295 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/wal
uuid: "065816551ea64988a938a5e2e163db38"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "42f3ab0a2e915996767a39b12e92ce58"
server_key_iv: "c26f36c77e594d02fe7baec90d99ccc2"
server_key_version: "encryptionkey@0"
I20260928 04:53:56.644376 6962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:56.654939 6962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:56.656971 6962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:56.657153 6962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:56.661449 6962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43647
I20260928 04:53:56.661505 7029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43647 every 8 connection(s)
I20260928 04:53:56.661895 6962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/data/info.pb
I20260928 04:53:56.663183 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:56.664834 7030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:53:56.667624 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 6962
Sep 28 04:53:56 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571236, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 04:53:56 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 04:53:56.676021 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.665690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33710 (local address 127.0.98.62:44343)
0928 04:53:56.665762 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:56.665764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:56.665954 (+ 190us) server_negotiation.cc:407] Connection header received
0928 04:53:56.666450 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:56.666451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:56.666461 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:56.666591 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:56.667460 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.668222 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.669300 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.669514 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.669607 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.672409 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:56.672411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:56.672412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:56.672420 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:56.673936 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.674475 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.674477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.674477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.674573 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.674928 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.674929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.674929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.675305 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:56.675390 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:56.675812 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 04:53:56.675915 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:53:56.676249 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.665612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.0.1:33710)
0928 04:53:56.665819 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:56.665862 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:56.666353 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:56.666673 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:56.666676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:56.667077 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:56.667309 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:56.667314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.668329 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.668330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:56.669191 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:56.669193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.669616 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.669616 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:56.669752 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.670317 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:56.670328 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:56.672309 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:56.674036 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:56.674046 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:56.674049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:56.674374 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:56.674648 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:56.674649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:56.674650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:56.674826 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:56.675406 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:56.675415 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:56.675619 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 04:53:56.675766 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 04:53:56.675959 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":123,"threads_started":1}
I20260928 04:53:56.676921 7030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:53:56.678629 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.668170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.0.98.61:43647)
0928 04:53:56.668386 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:56.668391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:56.668415 (+ 24us) server_negotiation.cc:407] Connection header received
0928 04:53:56.668437 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:56.668439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:56.668475 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:56.668561 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:56.669229 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.669974 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.671039 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.671583 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.671668 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.674768 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:56.674773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:56.674773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:56.674781 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:56.676793 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.677518 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.677521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.677521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.677587 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.677888 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.677890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.677890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.678133 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:56.678341 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:56.678395 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 04:53:56.678513 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":145,"threads_started":1}
Sep 28 04:53:56 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571236, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 04:53:56.684813 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.677307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47730 (local address 127.0.98.61:43647)
0928 04:53:56.677443 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:56.677444 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:56.677449 (+ 5us) server_negotiation.cc:407] Connection header received
0928 04:53:56.677461 (+ 12us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:56.677462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:56.677488 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:56.677554 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:56.677888 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.678852 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.679922 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.680136 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.680191 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.681996 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:56.681997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:56.681997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:56.682003 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:56.683867 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.684168 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.684169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.684169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.684220 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.684379 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.684380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.684380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.684474 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:56.684520 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:56.684651 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 04:53:56.684723 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
I20260928 04:53:56.684862 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.677245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43647 (local address 127.0.0.1:47730)
0928 04:53:56.677298 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:56.677309 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:56.677344 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:56.677558 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:56.677560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:56.677715 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:56.677764 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:56.677767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.679329 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.679330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:56.679838 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:56.679840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.680215 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.680215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:56.680259 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.680769 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:56.680773 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:56.681922 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:56.683944 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:56.683944 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:56.683945 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:56.684104 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:56.684276 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:56.684277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:56.684277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:56.684329 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:56.684544 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:56.684545 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:56.684582 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 04:53:56.684637 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 04:53:56.684771 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":15}
I20260928 04:53:56.686404 7030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: Bootstrap starting.
I20260928 04:53:56.687079 7030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:56.687404 7030 log.cc:824] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:56.688117 7030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: No bootstrap required, opened a new log
I20260928 04:53:56.689445 7030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } }
I20260928 04:53:56.689567 7030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:56.689589 7030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 065816551ea64988a938a5e2e163db38, State: Initialized, Role: FOLLOWER
I20260928 04:53:56.689730 7030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } }
I20260928 04:53:56.689982 7037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } }
I20260928 04:53:56.690052 7030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:56.690066 7037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 04:53:56.691725 7048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:53:56.691816 7048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:53:56.693500 6962 master_runner.cc:428] Detected that this master 065816551ea64988a938a5e2e163db38 is joining an existing cluster
I20260928 04:53:56.693573 6962 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:43647
I20260928 04:53:56.701884 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.694599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33714 (local address 127.0.98.62:44343)
0928 04:53:56.694731 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:56.694733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:56.694779 (+ 46us) server_negotiation.cc:407] Connection header received
0928 04:53:56.694889 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:56.694891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:56.694902 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:56.694932 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:56.695527 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.696033 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.697113 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.697368 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.697401 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.698824 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:56.698825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:56.698826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:56.698833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:56.700577 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.700974 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.700975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.700976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.701042 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.701286 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.701287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.701287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.701436 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:56.701501 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:56.701706 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 04:53:56.701789 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260928 04:53:56.701895 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.694563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.0.1:33714)
0928 04:53:56.694706 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:56.694720 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:56.694808 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:56.694996 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:56.694997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:56.695304 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:56.695427 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:56.695429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.696121 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.696122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:56.697015 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:56.697017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.697481 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.697482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:56.697523 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.698089 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:56.698095 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:56.698716 (+ 621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:56.700695 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:56.700696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:56.700697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:56.700892 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:56.701150 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:56.701151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:56.701151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:56.701215 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:56.701529 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:56.701530 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:56.701577 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 04:53:56.701672 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 04:53:56.701779 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":7,"thread_start_us":98,"threads_started":1}
I20260928 04:53:56.702412 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.694603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47738 (local address 127.0.98.61:43647)
0928 04:53:56.694643 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:56.694645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:56.694869 (+ 224us) server_negotiation.cc:407] Connection header received
0928 04:53:56.694909 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:56.694910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:56.694921 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:56.694975 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:56.695512 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.696275 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.697211 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.697526 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.697597 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.698888 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:56.698889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:56.698890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:56.698897 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:56.701297 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.701662 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.701663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.701663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.701721 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.701910 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.701910 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.701911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.702020 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:56.702057 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:56.702246 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 04:53:56.702324 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":20}
I20260928 04:53:56.702786 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.694675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43647 (local address 127.0.0.1:47738)
0928 04:53:56.694794 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:56.694803 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:56.694856 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:56.694994 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:56.694994 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:56.695302 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:56.695427 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:56.695429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.696357 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.696358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:56.697123 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:56.697124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.697612 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.697612 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:56.697682 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.698225 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:56.698230 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:56.698808 (+ 578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:56.701375 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:56.701375 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:56.701376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:56.701579 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:56.701795 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:56.701796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:56.701796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:56.701846 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:56.702095 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:56.702096 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:56.702137 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 04:53:56.702593 (+ 456us) client_negotiation.cc:240] Negotiation successful
0928 04:53:56.702686 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":16,"thread_start_us":84,"threads_started":1}
Sep 28 04:53:56 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571235, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 04:53:56.712743 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.704801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47740 (local address 127.0.98.61:43647)
0928 04:53:56.704857 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:56.704858 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:56.705048 (+ 190us) server_negotiation.cc:407] Connection header received
0928 04:53:56.705137 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:56.705138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:56.705146 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:56.705192 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:56.705701 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.706180 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.707359 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.707543 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.707592 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.709868 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:56.709870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:56.709871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:56.709879 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:56.711424 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.711860 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.711861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.711862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.711926 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.712148 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.712149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.712149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.712277 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:56.712353 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:56.712543 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 04:53:56.712639 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:53:56.712756 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.704714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43647 (local address 127.0.0.1:47740)
0928 04:53:56.704928 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:56.704978 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:56.705060 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:56.705203 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:56.705206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:56.705560 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:56.705626 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:56.705628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.706272 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.706273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:56.707269 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:56.707272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.707616 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.707617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:56.707702 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.708158 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:56.708163 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:56.709768 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:56.711543 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:56.711554 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:56.711556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:56.711779 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:56.711996 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:56.711996 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:56.711997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:56.712066 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:56.712345 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:56.712354 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:56.712414 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 04:53:56.712518 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 04:53:56.712647 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":107,"threads_started":1}
I20260928 04:53:56.713479 6496 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:43647: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "065816551ea64988a938a5e2e163db38"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 43647
}
attrs {
promote: true
}
}
dest_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857"
cas_config_opid_index: -1
I20260928 04:53:56.713810 6496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: true } }
I20260928 04:53:56.714304 7058 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 065816551ea64988a938a5e2e163db38 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: true } } }
I20260928 04:53:56.714694 7004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:56.715332 7058 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:43647
I20260928 04:53:56.715552 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: true } } }
I20260928 04:53:56.715624 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:56.715665 7004 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Refusing update from remote peer 55fc8a9c3dd54c1eaf1dece4e8993857: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 04:53:56.715998 6962 master.cc:537] Master@127.0.98.61:43647 shutting down...
I20260928 04:53:56.715986 7057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [LEADER]: Connected to new peer: Peer: permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 04:53:56.716846 6483 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 04:53:56.717401 6483 consensus_peers.cc:487] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 -> Peer 065816551ea64988a938a5e2e163db38 (127.0.98.61:43647): Couldn't send request to peer 065816551ea64988a938a5e2e163db38. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 04:53:56.719172 6962 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:53:56.719296 6962 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:56.719350 6962 tablet_replica.cc:330] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: stopping tablet replica
I20260928 04:53:56.732443 6962 master.cc:552] Master@127.0.98.61:43647 shutdown complete.
I20260928 04:53:56.732501 6962 master_runner.cc:305] Clearing existing system tablet
I20260928 04:53:56.732656 6962 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:56.732712 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:56.733384 6962 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.001s
I20260928 04:53:56.733906 7062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:56.734030 6962 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:56.734092 6962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:56.734115 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/wal
uuid: "065816551ea64988a938a5e2e163db38"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "42f3ab0a2e915996767a39b12e92ce58"
server_key_iv: "c26f36c77e594d02fe7baec90d99ccc2"
server_key_version: "encryptionkey@0"
I20260928 04:53:56.734920 6962 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:56.736528 6962 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 04:53:56.736699 6962 log.cc:1196] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 04:53:56.736910 6962 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: Deleting consensus metadata
I20260928 04:53:56.737037 6962 master_runner.cc:315] Copying system tablet from 127.0.98.62:44343
I20260928 04:53:56.737740 6962 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:44343
I20260928 04:53:56.744231 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.738166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.0.98.62:44343)
0928 04:53:56.738216 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:56.738218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:56.738299 (+ 81us) server_negotiation.cc:407] Connection header received
0928 04:53:56.738386 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:56.738387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:56.738398 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:56.738445 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:56.738890 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.739413 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.740187 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.740401 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.740489 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.741519 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:56.741521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:56.741521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:56.741529 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:56.742982 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.743351 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.743352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.743353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.743416 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.743628 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.743629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.743629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.743772 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:56.743858 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:56.744095 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 04:53:56.744165 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 04:53:56.744272 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.738080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.0.1:33730)
0928 04:53:56.738198 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:56.738214 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:56.738317 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:56.738448 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:56.738448 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:56.738736 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:56.738792 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:56.738795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.739516 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.739517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:56.740093 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:56.740095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.740493 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.740494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:56.740584 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.741113 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:56.741117 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:56.741418 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:56.743093 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:56.743094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:56.743094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:56.743276 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:56.743492 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:56.743492 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:56.743493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:56.743562 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:56.743869 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:56.743871 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:56.743929 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 04:53:56.744029 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 04:53:56.744144 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":68,"threads_started":1}
I20260928 04:53:56.744573 6517 tablet_copy_service.cc:140] P 55fc8a9c3dd54c1eaf1dece4e8993857: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 065816551ea64988a938a5e2e163db38 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33730)
I20260928 04:53:56.744670 6517 tablet_copy_service.cc:161] P 55fc8a9c3dd54c1eaf1dece4e8993857: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 065816551ea64988a938a5e2e163db38 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33730: session id = 065816551ea64988a938a5e2e163db38-00000000000000000000000000000000
I20260928 04:53:56.745379 6517 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 04:53:56.746084 6962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 04:53:56.747283 6962 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: tablet copy: Starting download of 0 data blocks...
I20260928 04:53:56.747457 6962 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: tablet copy: Starting download of 1 WAL segments...
I20260928 04:53:56.748411 6962 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 04:53:56.749185 6517 tablet_copy_service.cc:342] P 55fc8a9c3dd54c1eaf1dece4e8993857: Request end of tablet copy session 065816551ea64988a938a5e2e163db38-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33730
I20260928 04:53:56.749254 6517 tablet_copy_service.cc:433] P 55fc8a9c3dd54c1eaf1dece4e8993857: ending tablet copy session 065816551ea64988a938a5e2e163db38-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 065816551ea64988a938a5e2e163db38
I20260928 04:53:56.751796 6962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:56.753101 7069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:56.753145 7072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:56.753178 7070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:56.754683 6962 server_base.cc:1034] running on GCE node
I20260928 04:53:56.754825 6962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:56.755005 6962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:56.756172 6962 hybrid_clock.cc:648] HybridClock initialized: now 1790571236756160 us; error 31 us; skew 500 ppm
Sep 28 04:53:56 dist-test-slave-m39l krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571236, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:56.757611 6962 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 04:53:56.757898 6962 webserver.cc:460] Webserver started at http://127.0.98.61:42531/ using document root <none> and password file <none>
I20260928 04:53:56.758033 6962 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:56.758122 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:56.758805 6962 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:53:56.759382 7079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:56.759549 6962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:53:56.759610 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/wal
uuid: "065816551ea64988a938a5e2e163db38"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "42f3ab0a2e915996767a39b12e92ce58"
server_key_iv: "c26f36c77e594d02fe7baec90d99ccc2"
server_key_version: "encryptionkey@0"
I20260928 04:53:56.759666 6962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:56.786031 6962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:56.788261 6962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:56.791626 6962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43647
I20260928 04:53:56.791682 7131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43647 every 8 connection(s)
I20260928 04:53:56.792069 6962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-1/data/info.pb
I20260928 04:53:56.792833 7132 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:53:56.793447 7132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: Bootstrap starting.
I20260928 04:53:56.795810 7132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:53:56.796038 7132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: Bootstrap complete.
I20260928 04:53:56.796244 7132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: true } }
I20260928 04:53:56.796331 7132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 065816551ea64988a938a5e2e163db38, State: Initialized, Role: LEARNER
I20260928 04:53:56.796383 7132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: true } }
I20260928 04:53:56.796573 7132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:56.796531 7135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: true } } }
I20260928 04:53:56.796628 7135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [sys.catalog]: This master's current role is: LEARNER
I20260928 04:53:56.798255 7146 catalog_manager.cc:1259] Loaded cluster ID: 7c486fd6474b4aacbbe0b93916d5be95
I20260928 04:53:56.798321 7146 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: loading cluster ID for follower catalog manager: success
I20260928 04:53:56.799440 7146 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: acquiring CA information for follower catalog manager: success
I20260928 04:53:56.799746 7146 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:53:56.838529 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:56.831120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.0.98.61:43647)
0928 04:53:56.831268 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:56.831270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:56.831280 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:53:56.831317 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:56.831318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:56.831328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:56.831425 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:56.832050 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.832771 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.833572 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:56.833781 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:56.833886 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:56.835316 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:56.835318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:56.835319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:56.835336 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:56.837102 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.837569 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.837571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.837571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.837644 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:56.837875 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:56.837876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:56.837876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:56.838008 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:56.838133 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:56.838261 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 04:53:56.838373 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260928 04:53:56.870221 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43627
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44343,127.0.98.61:43647,127.0.98.60:43627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:53:56.955421 7150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:56.955673 7150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:56.955711 7150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:56.956934 7150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:53:56.957006 7150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:56.957032 7150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:56.957064 7150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:53:56.957090 7150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:53:56.958771 7150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44343,127.0.98.61:43647,127.0.98.60:43627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:56.959052 7150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:56.959302 7150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:56.964512 7156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:56.964516 7155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:56.964512 7158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:56.965087 7150 server_base.cc:1034] running on GCE node
I20260928 04:53:56.965353 7150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:56.965659 7150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:56.966833 7150 hybrid_clock.cc:648] HybridClock initialized: now 1790571236966816 us; error 57 us; skew 500 ppm
Sep 28 04:53:56 dist-test-slave-m39l krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571236, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:56.969679 7150 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 04:53:56.970269 7150 webserver.cc:460] Webserver started at http://127.0.98.60:41863/ using document root <none> and password file <none>
I20260928 04:53:56.970690 7150 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:56.970777 7150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:56.970952 7150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:53:56.972249 7150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/data/instance:
uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "b60a08787c74eaf9142f06a343759c79"
server_key_iv: "b010ef08ee83617c39174050dd340bf7"
server_key_version: "encryptionkey@0"
I20260928 04:53:56.972673 7150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/wal/instance:
uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "b60a08787c74eaf9142f06a343759c79"
server_key_iv: "b010ef08ee83617c39174050dd340bf7"
server_key_version: "encryptionkey@0"
I20260928 04:53:56.974244 7150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:53:56.975232 7165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:56.975425 7150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:53:56.975533 7150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/wal
uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "b60a08787c74eaf9142f06a343759c79"
server_key_iv: "b010ef08ee83617c39174050dd340bf7"
server_key_version: "encryptionkey@0"
I20260928 04:53:56.975608 7150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:56.990144 7150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:56.992468 7150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:56.992640 7150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:56.998224 7150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43627
I20260928 04:53:56.998605 7217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43627 every 8 connection(s)
I20260928 04:53:56.998739 7150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/data/info.pb
I20260928 04:53:57.003299 7218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:57.005198 7218 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:53:57.005582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 7150
Sep 28 04:53:57 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571236, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 04:53:57 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 04:53:57.017911 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.006204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.0.98.62:44343)
0928 04:53:57.006264 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.006267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.006608 (+ 341us) server_negotiation.cc:407] Connection header received
0928 04:53:57.007079 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.007081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.007093 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.007154 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.007928 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.008739 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.010155 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.010476 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.010581 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.013819 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.013820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.013821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.013828 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.016069 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.016724 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.016725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.016725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.016779 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.017087 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.017088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.017089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.017286 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.017357 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.017704 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.017796 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 04:53:57.018209 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.006153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.0.1:33732)
0928 04:53:57.006383 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.006426 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.006983 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.007190 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.007192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.007580 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:57.007841 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.007847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.008858 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.008859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.009965 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.009968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.010641 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.010660 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.010742 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.011744 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:57.011756 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:57.013682 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:57.016163 (+ 2481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.016169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.016205 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.016636 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.016846 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.016848 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.016848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.016995 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.017360 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:57.017367 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:57.017553 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.017639 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.017863 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":2,"thread_start_us":115,"threads_started":1}
I20260928 04:53:57.018466 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.006430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40240 (local address 127.0.98.60:43627)
0928 04:53:57.006660 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.006664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.006693 (+ 29us) server_negotiation.cc:407] Connection header received
0928 04:53:57.007001 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.007011 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.007055 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.007168 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.007914 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.009096 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.010087 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.010610 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.010704 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.014563 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.014572 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.014572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.014580 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.016575 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.017190 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.017193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.017194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.017252 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.017514 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.017515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.017515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.017752 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.017941 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.018291 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.018362 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
I20260928 04:53:57.018890 7218 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:53:57 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571236, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 04:53:57.027269 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.019159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43647 (local address 127.0.0.1:47760)
0928 04:53:57.019222 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.019231 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.019260 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.019455 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.019456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.019573 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:57.019623 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.019624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.020402 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.020403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.021197 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.021200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.021795 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.021796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.021904 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.022681 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:57.022688 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:57.024401 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:57.026192 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.026193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.026193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.026364 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.026632 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.026633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.026634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.026692 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.027013 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:57.027014 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:57.027061 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.027113 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.027198 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 04:53:57.027345 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.019258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47760 (local address 127.0.98.61:43647)
0928 04:53:57.019364 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.019366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.019373 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:53:57.019394 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.019394 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.019402 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.019452 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.019729 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.020307 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.021311 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.021637 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.021795 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.024506 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.024508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.024508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.024518 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.026096 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.026449 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.026451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.026451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.026558 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.026764 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.026765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.026766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.026893 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.026988 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.027187 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.027278 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 04:53:57.027860 7218 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43627 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:53:57 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571236, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 04:53:57.035122 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.028196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43627 (local address 127.0.0.1:40242)
0928 04:53:57.028245 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.028285 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.028336 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.028473 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.028474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.028622 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:57.028652 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.028654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.029306 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.029307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.029846 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.029848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.030252 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.030253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.030298 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.030808 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:57.030813 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:57.032084 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:57.033821 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.033821 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.033822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.033995 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.034354 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.034354 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.034355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.034419 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.034811 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:57.034812 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:57.034855 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.034924 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.035030 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 04:53:57.035254 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.028239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40242 (local address 127.0.98.60:43627)
0928 04:53:57.028310 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.028312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.028320 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:53:57.028399 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.028400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.028415 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.028470 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.028743 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.029224 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.029945 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.030164 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.030254 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.032252 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.032253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.032253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.032260 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.033712 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.034067 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.034068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.034069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.034169 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.034548 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.034549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.034549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.034735 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.034793 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.035098 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.035164 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 04:53:57.037046 7218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: Bootstrap starting.
I20260928 04:53:57.038084 7218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:57.038403 7218 log.cc:824] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:57.039350 7218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: No bootstrap required, opened a new log
I20260928 04:53:57.041389 7218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43627 } }
I20260928 04:53:57.041524 7218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:57.041548 7218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 920bfb0d0bb64d6baf003fbe1d5a99ff, State: Initialized, Role: FOLLOWER
I20260928 04:53:57.042658 7218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43627 } }
I20260928 04:53:57.043056 7218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:57.043059 7223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43627 } } }
I20260928 04:53:57.043207 7223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff [sys.catalog]: This master's current role is: FOLLOWER
W20260928 04:53:57.044904 7234 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:53:57.044958 7234 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:53:57.048218 7150 master_runner.cc:428] Detected that this master 920bfb0d0bb64d6baf003fbe1d5a99ff is joining an existing cluster
I20260928 04:53:57.048305 7150 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:43627
I20260928 04:53:57.056674 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.049337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.0.98.62:44343)
0928 04:53:57.049384 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.049386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.049534 (+ 148us) server_negotiation.cc:407] Connection header received
0928 04:53:57.049724 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.049725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.049739 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.049773 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.050338 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.051164 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.052412 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.052712 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.052752 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.053881 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.053883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.053883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.053891 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.055442 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.055818 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.055819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.055819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.055876 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.056117 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.056118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.056118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.056247 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.056328 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.056492 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.056575 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:53:57.056694 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.049265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.0.1:33742)
0928 04:53:57.049451 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.049460 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.049565 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.049859 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.049860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.050129 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:57.050227 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.050229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.051265 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.051265 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.052311 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.052313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.052818 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.052819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.052870 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.053396 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:57.053401 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:57.053762 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:57.055554 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.055555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.055555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.055741 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.055950 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.055951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.055951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.056033 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.056314 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:57.056320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:57.056365 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.056455 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.056576 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":122,"thread_start_us":90,"threads_started":1}
Sep 28 04:53:57 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571235, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 04:53:57.070003 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 6560
I20260928 04:53:57.075457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40899
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44343
--webserver_port=36849
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44343,127.0.98.61:43647
--tserver_master_addrs=127.0.98.62:44343,127.0.98.61:43647,127.0.98.60:43627 with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:53:57.087193 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.049563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47766 (local address 127.0.98.61:43647)
0928 04:53:57.049604 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.049605 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.049866 (+ 261us) server_negotiation.cc:407] Connection header received
0928 04:53:57.049914 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.049915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.049925 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.049979 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.050392 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.051168 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.052381 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.052666 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.052706 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.084102 (+ 31396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.084104 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.084104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.084113 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.085732 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.086307 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.086308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.086309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.086367 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.086586 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.086587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.086587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.086737 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.086780 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.086975 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.087087 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:53:57.087638 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.049498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43647 (local address 127.0.0.1:47766)
0928 04:53:57.049794 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.049804 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.049858 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.050027 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.050028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.050251 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:57.050308 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.050310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.051265 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.051265 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.052275 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.052277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.052799 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.052799 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.052864 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.083181 (+ 30317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:57.083189 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:57.083960 (+ 771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:57.085839 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.085841 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.085842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.086197 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.086437 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.086437 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.086438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.086517 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.086824 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:57.086825 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:57.086871 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.087398 (+ 527us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.087515 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260928 04:53:57.088248 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.059327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43627 (local address 127.0.0.1:40254)
0928 04:53:57.059401 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.059413 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.059462 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.059836 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.059838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.060009 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:57.060050 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.060052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.060798 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.060799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.061451 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.061454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.061886 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.061887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.061953 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.062399 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:57.062405 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:57.063774 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:57.086708 (+ 22934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.086710 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.086711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.086953 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.087167 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.087168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.087168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.087223 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.087845 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:57.087846 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:57.087905 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.087981 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.088123 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 04:53:57.088371 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.059422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40254 (local address 127.0.98.60:43627)
0928 04:53:57.059574 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.059575 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.059658 (+ 83us) server_negotiation.cc:407] Connection header received
0928 04:53:57.059732 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.059732 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.059745 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.059832 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.060138 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.060677 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.061541 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.061787 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.061883 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.063877 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.063878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.063878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.063885 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.086601 (+ 22716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.087042 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.087043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.087043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.087099 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.087479 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.087480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.087480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.087623 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.087689 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.088176 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.088264 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":104,"threads_started":1}
I20260928 04:53:57.088792 6496 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:43627: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 43627
}
attrs {
promote: true
}
}
dest_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857"
cas_config_opid_index: 5
I20260928 04:53:57.089044 6496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: true } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } }
I20260928 04:53:57.089427 7058 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 920bfb0d0bb64d6baf003fbe1d5a99ff (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: true } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } } }
I20260928 04:53:57.090824 7191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:57.091912 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.049693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43627 (local address 127.0.0.1:40250)
0928 04:53:57.049949 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.049957 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.049995 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.050140 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.050140 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.050409 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:57.050467 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.050469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.051363 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.051364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.052300 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.052302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.052779 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.052780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.052835 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.084375 (+ 31540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:57.084379 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:57.088617 (+ 4238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:57.090763 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.090764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.090764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.091024 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.091219 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.091220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.091220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.091269 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.091536 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:57.091537 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:57.091577 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.091675 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.091800 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":89,"thread_start_us":53,"threads_started":1}
I20260928 04:53:57.092298 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.049720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.0.98.60:43627)
0928 04:53:57.049756 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.049758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.050040 (+ 282us) server_negotiation.cc:407] Connection header received
0928 04:53:57.050056 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.050056 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.050065 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.050103 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.050591 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.051281 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.052389 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.052673 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.052730 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.088700 (+ 35970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.088701 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.088701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.088708 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.090679 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.091102 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.091103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.091103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.091150 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.091326 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.091326 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.091326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.091459 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.091514 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.092096 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.092175 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:53:57.094687 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.089576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.0.98.61:43647)
0928 04:53:57.089753 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.089755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.089762 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:53:57.089782 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.089783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.089806 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.089873 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:57.090063 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.092734 (+ 2671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.093856 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.094254 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.094381 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.094468 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.094571 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.094627 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260928 04:53:57.094691 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.089483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43647 (local address 127.0.0.1:47772)
0928 04:53:57.089621 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.089631 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.089677 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.089930 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.089931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.089933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:57.089984 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.089986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.092831 (+ 2845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.092833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.093747 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.093750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.094385 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.094386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.094450 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.094479 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.094564 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.094617 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260928 04:53:57.095074 7106 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 LEARNER]: Refusing update from remote peer 55fc8a9c3dd54c1eaf1dece4e8993857: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 04:53:57.095340 7057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [LEADER]: Connected to new peer: Peer: permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 04:53:57.096055 7106 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 920bfb0d0bb64d6baf003fbe1d5a99ff (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: true } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } } }
I20260928 04:53:57.122268 7058 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:43627
I20260928 04:53:57.122400 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: true } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } } }
I20260928 04:53:57.122552 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:57.123474 7135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55fc8a9c3dd54c1eaf1dece4e8993857. Latest consensus state: current_term: 1 leader_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: true } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } } }
I20260928 04:53:57.123598 7135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [sys.catalog]: This master's current role is: LEARNER
I20260928 04:53:57.123644 7135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: true } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } } }
I20260928 04:53:57.123703 7135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [sys.catalog]: This master's current role is: LEARNER
I20260928 04:53:57.123822 7058 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857: attempting to promote NON_VOTER 065816551ea64988a938a5e2e163db38 to VOTER
I20260928 04:53:57.124015 7058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: false } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } }
I20260928 04:53:57.124655 7106 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 LEARNER]: Refusing update from remote peer 55fc8a9c3dd54c1eaf1dece4e8993857: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 04:53:57.124833 7058 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [LEADER]: Connected to new peer: Peer: permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 04:53:57.125473 7191 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff [term 1 FOLLOWER]: Refusing update from remote peer 55fc8a9c3dd54c1eaf1dece4e8993857: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 04:53:57.125535 7150 master.cc:537] Master@127.0.98.60:43627 shutting down...
I20260928 04:53:57.125581 7192 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff [term 1 FOLLOWER]: Refusing update from remote peer 55fc8a9c3dd54c1eaf1dece4e8993857: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 04:53:57.125795 7057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [LEADER]: Connected to new peer: Peer: permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:57.126055 7058 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 065816551ea64988a938a5e2e163db38 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: false } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } } }
I20260928 04:53:57.127156 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: false } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } } }
I20260928 04:53:57.127230 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [sys.catalog]: This master's current role is: LEADER
W20260928 04:53:57.127504 6484 consensus_peers.cc:487] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 -> Peer 920bfb0d0bb64d6baf003fbe1d5a99ff (127.0.98.60:43627): Couldn't send request to peer 920bfb0d0bb64d6baf003fbe1d5a99ff. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 04:53:57.127521 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: false } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } } }
I20260928 04:53:57.127727 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857 [sys.catalog]: This master's current role is: LEADER
I20260928 04:53:57.127727 7106 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 065816551ea64988a938a5e2e163db38 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: false } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } } }
I20260928 04:53:57.128808 7246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: false } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } } }
I20260928 04:53:57.129000 7246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:53:57.128986 7150 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:53:57.129078 7150 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:57.129185 7150 tablet_replica.cc:330] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: stopping tablet replica
I20260928 04:53:57.144620 7150 master.cc:552] Master@127.0.98.60:43627 shutdown complete.
I20260928 04:53:57.144691 7150 master_runner.cc:305] Clearing existing system tablet
I20260928 04:53:57.144878 7150 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:57.144928 7150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:57.145831 7150 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:53:57.146385 7249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:57.146584 7150 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:57.146649 7150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:53:57.146680 7150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/wal
uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "b60a08787c74eaf9142f06a343759c79"
server_key_iv: "b010ef08ee83617c39174050dd340bf7"
server_key_version: "encryptionkey@0"
I20260928 04:53:57.147449 7150 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:57.177894 7150 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 04:53:57.178083 7150 log.cc:1196] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 04:53:57.178352 7150 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: Deleting consensus metadata
I20260928 04:53:57.178580 7150 master_runner.cc:315] Copying system tablet from 127.0.98.62:44343
I20260928 04:53:57.179298 7150 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:44343
I20260928 04:53:57.186640 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.179663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.0.1:33744)
0928 04:53:57.179814 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.179825 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.179919 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.180104 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.180105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.180309 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:57.180383 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.180385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.181192 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.181194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.182150 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.182153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.182633 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.182634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.182707 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.183274 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:57.183278 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:57.183638 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:57.185419 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.185420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.185420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.185616 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.185849 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.185850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.185850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.185921 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.186206 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:57.186207 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:57.186269 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.186364 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.186478 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260928 04:53:57.186628 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.179714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33744 (local address 127.0.98.62:44343)
0928 04:53:57.179770 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.179772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.179902 (+ 130us) server_negotiation.cc:407] Connection header received
0928 04:53:57.179995 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.179996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.180008 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.180098 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.180504 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.181090 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.182267 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.182463 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.182632 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.183764 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.183766 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.183767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.183774 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.185279 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.185709 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.185711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.185711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.185777 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.186009 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.186010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.186011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.186131 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.186217 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.186389 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.186526 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:53:57.187343 6517 tablet_copy_service.cc:140] P 55fc8a9c3dd54c1eaf1dece4e8993857: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 920bfb0d0bb64d6baf003fbe1d5a99ff ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33744)
I20260928 04:53:57.187430 6517 tablet_copy_service.cc:161] P 55fc8a9c3dd54c1eaf1dece4e8993857: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 920bfb0d0bb64d6baf003fbe1d5a99ff at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33744: session id = 920bfb0d0bb64d6baf003fbe1d5a99ff-00000000000000000000000000000000
I20260928 04:53:57.187907 6517 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 04:53:57.188488 7150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 04:53:57.189558 7150 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: tablet copy: Starting download of 0 data blocks...
I20260928 04:53:57.189730 7150 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: tablet copy: Starting download of 1 WAL segments...
I20260928 04:53:57.190531 7150 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 04:53:57.191483 6517 tablet_copy_service.cc:342] P 55fc8a9c3dd54c1eaf1dece4e8993857: Request end of tablet copy session 920bfb0d0bb64d6baf003fbe1d5a99ff-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33744
I20260928 04:53:57.191570 6517 tablet_copy_service.cc:433] P 55fc8a9c3dd54c1eaf1dece4e8993857: ending tablet copy session 920bfb0d0bb64d6baf003fbe1d5a99ff-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 920bfb0d0bb64d6baf003fbe1d5a99ff
I20260928 04:53:57.194346 7150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:57.195900 7256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:57.195977 7259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:57.196179 7150 server_base.cc:1034] running on GCE node
W20260928 04:53:57.196269 7257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:57.196396 7150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:57.196537 7150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:57.197661 7150 hybrid_clock.cc:648] HybridClock initialized: now 1790571237197642 us; error 36 us; skew 500 ppm
Sep 28 04:53:57 dist-test-slave-m39l krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:57.199270 7150 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 04:53:57.199604 7150 webserver.cc:460] Webserver started at http://127.0.98.60:42527/ using document root <none> and password file <none>
I20260928 04:53:57.199739 7150 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:57.199800 7150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:57.200570 7150 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:57.201159 7269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:57.201292 7150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:53:57.201332 7150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/wal
uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "b60a08787c74eaf9142f06a343759c79"
server_key_iv: "b010ef08ee83617c39174050dd340bf7"
server_key_version: "encryptionkey@0"
I20260928 04:53:57.201399 7150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260928 04:53:57.202766 7244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:57.203038 7244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:57.203089 7244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:57.204845 7244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:57.204912 7244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:57.204955 7244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:53:57.207286 7244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36849
--tserver_master_addrs=127.0.98.62:44343,127.0.98.61:43647,127.0.98.60:43627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:57.207595 7244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:57.207858 7244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:53:57.212666 7150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:57.215170 7150 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260928 04:53:57.215801 7278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:57.216682 7276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:57.218137 7244 server_base.cc:1034] running on GCE node
I20260928 04:53:57.219810 7150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43627
W20260928 04:53:57.220189 7275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:57.220396 7150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/master-2/data/info.pb
I20260928 04:53:57.220496 7244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:57.220829 7244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:57.221053 7327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43627 every 8 connection(s)
I20260928 04:53:57.221596 7328 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:53:57.222031 7244 hybrid_clock.cc:648] HybridClock initialized: now 1790571237221978 us; error 68 us; skew 500 ppm
I20260928 04:53:57.222325 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: Bootstrap starting.
Sep 28 04:53:57 dist-test-slave-m39l krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:57.225745 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:53:57.225788 7244 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 04:53:57.225996 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: Bootstrap complete.
I20260928 04:53:57.226215 7328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: false } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } }
I20260928 04:53:57.226333 7328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 920bfb0d0bb64d6baf003fbe1d5a99ff, State: Initialized, Role: LEARNER
I20260928 04:53:57.226390 7328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: false } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } }
I20260928 04:53:57.226457 7244 webserver.cc:460] Webserver started at http://127.0.98.1:36849/ using document root <none> and password file <none>
I20260928 04:53:57.226615 7328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:53:57.226784 7244 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:57.226867 7244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:57.229072 7346 catalog_manager.cc:1259] Loaded cluster ID: 7c486fd6474b4aacbbe0b93916d5be95
I20260928 04:53:57.229147 7346 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: loading cluster ID for follower catalog manager: success
I20260928 04:53:57.230293 7346 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: acquiring CA information for follower catalog manager: success
I20260928 04:53:57.229097 7335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: false } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } } }
I20260928 04:53:57.230648 7346 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:53:57.230654 7335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 920bfb0d0bb64d6baf003fbe1d5a99ff [sys.catalog]: This master's current role is: LEARNER
I20260928 04:53:57.231060 7244 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 04:53:57.231812 7348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:57.232028 7244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:57.232127 7244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "db43418eb20a4d5ca92826ab4c6db24e"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "c7b3b275aa0f201d8f141c5f2d33c169"
server_key_iv: "0e8293ff533eb45e00c93aef385861f9"
server_key_version: "encryptionkey@0"
I20260928 04:53:57.232609 7244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:57.241241 7244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:57.243301 7244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:57.243418 7244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:57.243611 7244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:57.243942 7244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:57.243996 7244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:57.244020 7244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:57.244079 7244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:57.249996 7244 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40899
I20260928 04:53:57.250077 7461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40899 every 8 connection(s)
I20260928 04:53:57.250460 7244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:53:57.251848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 7244
I20260928 04:53:57.252000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 6694
I20260928 04:53:57.259137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41419
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44343
--webserver_port=34307
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44343,127.0.98.61:43647
--tserver_master_addrs=127.0.98.62:44343,127.0.98.61:43647,127.0.98.60:43627 with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:53:57 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 04:53:57 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 04:53:57 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 04:53:57.265553 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.251581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41191 (local address 127.0.98.62:44343)
0928 04:53:57.251629 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.251631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.251903 (+ 272us) server_negotiation.cc:407] Connection header received
0928 04:53:57.253539 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.253540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.253554 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.253597 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.254800 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.255595 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.257251 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.257626 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.257667 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.261554 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.261557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.261557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.261567 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.263642 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.264290 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.264292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.264292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.264359 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.264693 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.264694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.264694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.264848 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.264895 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.265311 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.265419 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:53:57.265599 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.251319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49805 (local address 127.0.98.60:43627)
0928 04:53:57.251547 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.251549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.251947 (+ 398us) server_negotiation.cc:407] Connection header received
0928 04:53:57.253585 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.253587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.253610 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.253673 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.254799 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.255957 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.257244 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.257571 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.257632 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.262218 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.262220 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.262220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.262230 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.264068 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.264519 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.264521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.264521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.264601 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.264846 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.264847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.264848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.265007 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.265056 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.265339 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.265457 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":60,"threads_started":1}
I20260928 04:53:57.265743 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.251294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54957 (local address 127.0.98.61:43647)
0928 04:53:57.251349 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.251352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.251865 (+ 513us) server_negotiation.cc:407] Connection header received
0928 04:53:57.253539 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.253540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.253554 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.253590 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.254801 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.255594 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.257251 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.257561 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.257604 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.262184 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.262187 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.262187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.262197 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.264512 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.264893 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.264895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.264895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.264954 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.265159 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.265160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.265161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.265309 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.265351 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.265541 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.265629 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 04:53:57.266371 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.251260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43647 (local address 127.0.98.1:54957)
0928 04:53:57.251751 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.251772 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.253400 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.253749 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.253754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.254322 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:57.254681 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.254687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.255738 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.255739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.257140 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.257143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.257704 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.257705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.257867 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.258781 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:57.258785 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:57.261758 (+ 2973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:57.264597 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.264597 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.264598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.264804 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.265023 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.265024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.265024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.265075 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.265399 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:57.265400 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:57.265437 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.265836 (+ 399us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.266108 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":283,"spinlock_wait_cycles":7552,"thread_start_us":113,"threads_started":1}
I20260928 04:53:57.266374 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.251260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43627 (local address 127.0.98.1:49805)
0928 04:53:57.251711 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.251765 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.253402 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.253749 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.253754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.254359 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:57.254681 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.254688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.256058 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.256059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.257142 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.257144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.257700 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.257701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.257867 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.258893 (+ 1026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:57.258900 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:57.262093 (+ 3193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:57.264179 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.264180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.264181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.264422 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.264699 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.264700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.264701 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.264769 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.265109 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:57.265110 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:57.265160 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.265760 (+ 600us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.266009 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":49,"thread_start_us":96,"threads_started":1}
I20260928 04:53:57.266968 7464 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43647
I20260928 04:53:57.267037 7462 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43627
I20260928 04:53:57.267079 7464 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:57.267112 7462 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:57.267190 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.251494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:41191)
0928 04:53:57.251843 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.251853 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.253410 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.253749 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.253754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.254372 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:57.254681 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.254687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.255738 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.255739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.257140 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.257143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.257740 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.257740 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.257867 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.258747 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:57.258760 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:57.261379 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:57.263759 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.263773 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.263776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.264180 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.264438 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.264439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.264439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.264613 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.264943 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:57.264950 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:57.265149 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.266352 (+ 1203us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.266972 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":52,"thread_start_us":55,"threads_started":1}
I20260928 04:53:57.267316 7462 heartbeater.cc:511] Master 127.0.98.60:43627 requested a full tablet report, sending...
I20260928 04:53:57.267318 7464 heartbeater.cc:511] Master 127.0.98.61:43647 requested a full tablet report, sending...
I20260928 04:53:57.267623 7463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44343
I20260928 04:53:57.267673 7463 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:57.267729 7463 heartbeater.cc:511] Master 127.0.98.62:44343 requested a full tablet report, sending...
I20260928 04:53:57.267868 7285 ts_manager.cc:194] Registered new tserver with Master: db43418eb20a4d5ca92826ab4c6db24e (127.0.98.1:40899)
I20260928 04:53:57.267899 7096 ts_manager.cc:194] Registered new tserver with Master: db43418eb20a4d5ca92826ab4c6db24e (127.0.98.1:40899)
I20260928 04:53:57.267969 6496 ts_manager.cc:194] Re-registered known tserver with Master: db43418eb20a4d5ca92826ab4c6db24e (127.0.98.1:40899)
I20260928 04:53:57.268411 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41191
W20260928 04:53:57.351356 7470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:57.351616 7470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:57.351657 7470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:57.352869 7470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:57.352945 7470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:57.352994 7470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:53:57.354689 7470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34307
--tserver_master_addrs=127.0.98.62:44343,127.0.98.61:43647,127.0.98.60:43627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:57.354978 7470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:57.355242 7470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:57.360759 7475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:57.360759 7476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:57.360924 7478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:57.360924 7470 server_base.cc:1034] running on GCE node
I20260928 04:53:57.361202 7470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:57.361415 7470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:57.362633 7470 hybrid_clock.cc:648] HybridClock initialized: now 1790571237362620 us; error 28 us; skew 500 ppm
Sep 28 04:53:57 dist-test-slave-m39l krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:57.365311 7470 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 04:53:57.365828 7470 webserver.cc:460] Webserver started at http://127.0.98.2:34307/ using document root <none> and password file <none>
I20260928 04:53:57.366086 7470 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:57.366171 7470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:57.367689 7470 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:57.368453 7485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:57.368681 7470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:57.368854 7470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "62ddf5590453482a9ada0d96856c81da"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "625fa203024fd4a7d9036c4ee19a26df"
server_key_iv: "01182343f6f27fbb989c030f506730e6"
server_key_version: "encryptionkey@0"
I20260928 04:53:57.369271 7470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:57.383442 7470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:57.385957 7470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:57.386139 7470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:57.386397 7470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:57.386756 7470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:57.386814 7470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:57.386860 7470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:57.386889 7470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:57.393071 7470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41419
I20260928 04:53:57.393244 7598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41419 every 8 connection(s)
I20260928 04:53:57.393555 7470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:53:57.395475 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 7470
I20260928 04:53:57.395675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 6828
I20260928 04:53:57.403002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36755
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44343
--webserver_port=32861
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44343,127.0.98.61:43647
--tserver_master_addrs=127.0.98.62:44343,127.0.98.61:43647,127.0.98.60:43627 with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:53:57 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 04:53:57 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 04:53:57 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 04:53:57.407395 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.394442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48325 (local address 127.0.98.61:43647)
0928 04:53:57.394526 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.394528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.397053 (+ 2525us) server_negotiation.cc:407] Connection header received
0928 04:53:57.397186 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.397187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.397201 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.397250 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.397880 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.398785 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.400353 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.400700 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.400751 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.403990 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.403992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.403992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.404001 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.405718 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.406147 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.406148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.406149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.406219 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.406575 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.406576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.406577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.406742 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.406789 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.407209 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.407299 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:53:57.408380 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.394382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43647 (local address 127.0.98.2:48325)
0928 04:53:57.395247 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.395257 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.397095 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.397339 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.397340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.397525 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:57.397722 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.397729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.398887 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.398888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.400229 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.400233 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.400857 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.400858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.401046 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.401718 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:57.401730 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:57.403873 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:57.405818 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.405828 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.405832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.406066 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.406295 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.406296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.406296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.406448 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.406839 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:57.406845 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:57.407044 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.407882 (+ 838us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.408093 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":285,"thread_start_us":71,"threads_started":1}
I20260928 04:53:57.409165 7601 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43647
I20260928 04:53:57.409271 7601 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:57.409330 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.394686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40975 (local address 127.0.98.62:44343)
0928 04:53:57.395161 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.395164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.395179 (+ 15us) server_negotiation.cc:407] Connection header received
0928 04:53:57.396557 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.396559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.396576 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.396648 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.397851 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.398776 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.401515 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.401961 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.402017 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.405340 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.405342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.405342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.405351 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.407161 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.408095 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.408097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.408097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.408163 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.408377 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.408378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.408379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.408880 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.408925 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.409135 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.409240 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 04:53:57.409607 7601 heartbeater.cc:511] Master 127.0.98.61:43647 requested a full tablet report, sending...
I20260928 04:53:57.409761 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.394377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:40975)
0928 04:53:57.394697 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.394745 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.396425 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.396809 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.396814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.397355 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:57.397722 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.397729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.398887 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.398888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.401334 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.401337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.402108 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.402109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.402175 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.403420 (+ 1245us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:57.403426 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:57.405215 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:57.407564 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.407565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.407565 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.407776 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.408252 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.408253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.408253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.408307 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.408975 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:57.408976 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:57.409019 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.409505 (+ 486us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.409624 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":1084,"thread_start_us":101,"threads_started":1}
I20260928 04:53:57.410200 7600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44343
I20260928 04:53:57.410233 7096 ts_manager.cc:194] Registered new tserver with Master: 62ddf5590453482a9ada0d96856c81da (127.0.98.2:41419)
I20260928 04:53:57.410255 7600 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:57.410336 7600 heartbeater.cc:511] Master 127.0.98.62:44343 requested a full tablet report, sending...
I20260928 04:53:57.410629 6496 ts_manager.cc:194] Re-registered known tserver with Master: 62ddf5590453482a9ada0d96856c81da (127.0.98.2:41419)
I20260928 04:53:57.410967 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40975
I20260928 04:53:57.411129 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.394578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52443 (local address 127.0.98.60:43627)
0928 04:53:57.394620 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.394622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.395303 (+ 681us) server_negotiation.cc:407] Connection header received
0928 04:53:57.396679 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.396681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.396723 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.396756 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.397840 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.398741 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.403030 (+ 4289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.403381 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.403453 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.407553 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.407556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.407556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.407566 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.409445 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.410136 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.410137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.410137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.410211 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.410510 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.410511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.410512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.410645 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.410702 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.410941 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.411032 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:53:57.411437 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.394379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43627 (local address 127.0.98.2:52443)
0928 04:53:57.395189 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.395202 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.396593 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.396840 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.396841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.397359 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:57.397722 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.397730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.398877 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.398878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.402926 (+ 4048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.402929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.403983 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.403984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.404061 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.404725 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:57.404730 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:57.407426 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:57.409790 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.409791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.409792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.410028 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.410323 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.410324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.410324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.410409 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.410771 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:57.410772 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:57.410815 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.410900 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.411056 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":2117,"thread_start_us":216,"threads_started":1}
I20260928 04:53:57.411871 7599 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43627
I20260928 04:53:57.411926 7599 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:57.411999 7599 heartbeater.cc:511] Master 127.0.98.60:43627 requested a full tablet report, sending...
I20260928 04:53:57.412298 7285 ts_manager.cc:194] Registered new tserver with Master: 62ddf5590453482a9ada0d96856c81da (127.0.98.2:41419)
W20260928 04:53:57.491214 7606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:53:57.491434 7606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:53:57.491460 7606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:53:57.492642 7606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:53:57.492695 7606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:53:57.492722 7606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:53:57.494232 7606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=32861
--tserver_master_addrs=127.0.98.62:44343,127.0.98.61:43647,127.0.98.60:43627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:53:57.494552 7606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:53:57.494834 7606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:53:57.500660 7612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:53:57.500778 7614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:57.500813 7606 server_base.cc:1034] running on GCE node
W20260928 04:53:57.500932 7611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:53:57.501158 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:53:57.501443 7606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:53:57.502602 7606 hybrid_clock.cc:648] HybridClock initialized: now 1790571237502581 us; error 52 us; skew 500 ppm
Sep 28 04:53:57 dist-test-slave-m39l krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:53:57.505461 7606 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 04:53:57.506067 7606 webserver.cc:460] Webserver started at http://127.0.98.3:32861/ using document root <none> and password file <none>
I20260928 04:53:57.506328 7606 fs_manager.cc:359] Metadata directory not provided
I20260928 04:53:57.506428 7606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:53:57.508066 7606 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:57.508836 7621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:57.509042 7606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:53:57.509155 7606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "ea50a1c39d174993ae777a27876c8403"
format_stamp: "Formatted at 2026-09-28 04:53:56 on dist-test-slave-m39l"
server_key: "25dd70080156fc373f4b02c186b7933b"
server_key_iv: "a1348cb7ce916b12ab69a1e2532008d5"
server_key_version: "encryptionkey@0"
I20260928 04:53:57.509570 7606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:53:57.526185 7606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:53:57.529353 7606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:53:57.529572 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:53:57.529867 7606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:53:57.530272 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:53:57.530337 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:57.530390 7606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:53:57.530422 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:53:57.537545 7606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36755
I20260928 04:53:57.537592 7734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36755 every 8 connection(s)
I20260928 04:53:57.538049 7606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:53:57.539428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 7606
Sep 28 04:53:57 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 04:53:57 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:53:57.551636 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.541359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47782 (local address 127.0.98.61:43647)
0928 04:53:57.541630 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.541632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.541743 (+ 111us) server_negotiation.cc:407] Connection header received
0928 04:53:57.541769 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.541770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.541782 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.541839 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.542675 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.543357 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.544441 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.544809 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.544858 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.547475 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.547477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.547477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.547486 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.549913 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.550372 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.550373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.550374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.550440 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.550692 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.550693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.550693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.550833 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.550924 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.551420 (+ 496us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.551516 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":249,"thread_start_us":57,"threads_started":1}
I20260928 04:53:57.552929 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.541513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.0.98.60:43627)
0928 04:53:57.541682 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.541684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.542060 (+ 376us) server_negotiation.cc:407] Connection header received
0928 04:53:57.542093 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.542094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.542109 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.542172 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.543052 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.543984 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.545156 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.545495 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.545639 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.547473 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.547475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.547475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.547485 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.549912 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.551751 (+ 1839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.551753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.551753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.551823 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.551998 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.551999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.551999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.552127 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.552252 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.552701 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.552802 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":59,"threads_started":1}
I20260928 04:53:57.553690 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.540716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39223 (local address 127.0.98.62:44343)
0928 04:53:57.540775 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.540778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.540791 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:53:57.540823 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.540824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.540836 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.540908 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.542226 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.543135 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.544468 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.544806 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.544852 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.548408 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.548411 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.548411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.548422 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.550411 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.552074 (+ 1663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.552075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.552076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.552131 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.552483 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.552484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.552484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.552628 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.552670 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.553497 (+ 827us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.553580 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 04:53:57 dist-test-slave-m39l krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 04:53:57.554595 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.538920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:39223)
0928 04:53:57.539456 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.539541 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.540292 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.540980 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.540982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.542003 (+ 1021us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:57.542094 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.542096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.543250 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.543251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.544356 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.544358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.544936 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.544937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.545066 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.546407 (+ 1341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:57.546412 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:57.548307 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:57.551303 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.551314 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.551317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.551987 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.552215 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.552216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.552217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.552394 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.552858 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:57.552865 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:57.553326 (+ 461us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.554109 (+ 783us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.554310 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":86,"threads_started":1}
I20260928 04:53:57.554708 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.539318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51067 (local address 127.0.98.61:43647)
0928 04:53:57.539361 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.539362 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.540531 (+ 1169us) server_negotiation.cc:407] Connection header received
0928 04:53:57.540591 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.540592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.540607 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.540664 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.542026 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.542828 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.544445 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.544763 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.544817 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.548818 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.548820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.548820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.548829 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.551209 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.552700 (+ 1491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.552701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.552701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.552755 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.553991 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.553992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.553992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.554140 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.554190 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.554444 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.554585 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":23}
I20260928 04:53:57.555115 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.538920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43647 (local address 127.0.98.3:51067)
0928 04:53:57.540450 (+ 1530us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.540459 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.540527 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.540851 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.540858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.541431 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:57.541776 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.541784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.542964 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.542964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.544346 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.544349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.544919 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.544920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.545066 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.545593 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:57.545605 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:57.548679 (+ 3074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:57.552288 (+ 3609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.552289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.552289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.552490 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.552819 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.552820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.552820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.552892 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.554254 (+ 1362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:57.554255 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:57.554305 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.554854 (+ 549us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.554983 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1498,"mutex_wait_us":5,"thread_start_us":74,"threads_started":1}
I20260928 04:53:57.555418 7736 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44343
I20260928 04:53:57.555454 7737 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43647
I20260928 04:53:57.555544 7736 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:57.555544 7737 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:57.555790 7737 heartbeater.cc:511] Master 127.0.98.61:43647 requested a full tablet report, sending...
I20260928 04:53:57.555790 7736 heartbeater.cc:511] Master 127.0.98.62:44343 requested a full tablet report, sending...
I20260928 04:53:57.556236 6496 ts_manager.cc:194] Re-registered known tserver with Master: ea50a1c39d174993ae777a27876c8403 (127.0.98.3:36755)
I20260928 04:53:57.556274 7096 ts_manager.cc:194] Registered new tserver with Master: ea50a1c39d174993ae777a27876c8403 (127.0.98.3:36755)
I20260928 04:53:57.556766 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39223
I20260928 04:53:57.557545 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.539183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58263 (local address 127.0.98.60:43627)
0928 04:53:57.539274 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.539277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.540997 (+ 1720us) server_negotiation.cc:407] Connection header received
0928 04:53:57.541095 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.541096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.541108 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.541138 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.545637 (+ 4499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.546407 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.550278 (+ 3871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.550624 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.550672 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.554349 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.554350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.554351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.554357 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.556205 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.556643 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.556644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.556644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.556710 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.556940 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.556941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.556942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.557082 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.557140 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.557365 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.557452 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 04:53:57.557897 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.538920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43627 (local address 127.0.98.3:58263)
0928 04:53:57.540925 (+ 2005us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.540936 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.541024 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.541208 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.541209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.545415 (+ 4206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:53:57.545483 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.545486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.547004 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.547005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.550178 (+ 3173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.550181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.550774 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.550775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.550843 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.551269 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:53:57.551275 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:53:57.554247 (+ 2972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:53:57.556307 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.556308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.556309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.556554 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.556805 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:53:57.556806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:53:57.556807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:53:57.556878 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:53:57.557181 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:53:57.557181 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:53:57.557240 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.557639 (+ 399us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.557821 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1661,"mutex_wait_us":65,"thread_start_us":76,"threads_started":1}
I20260928 04:53:57.558351 7735 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43627
I20260928 04:53:57.558423 7735 heartbeater.cc:464] Registering TS with master...
I20260928 04:53:57.558531 7735 heartbeater.cc:511] Master 127.0.98.60:43627 requested a full tablet report, sending...
I20260928 04:53:57.558828 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.540908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.0.98.62:44343)
0928 04:53:57.541555 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.541558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.541692 (+ 134us) server_negotiation.cc:407] Connection header received
0928 04:53:57.541735 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.541736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.541749 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.541854 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.550029 (+ 8175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.550940 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.552015 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.552377 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.552420 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.553945 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.553946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.553947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.553955 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.557071 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.557443 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.557444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.557444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.557501 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.557856 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.557857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.557857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.558019 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.558121 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.558648 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.558742 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":80,"threads_started":1}
I20260928 04:53:57.558841 7285 ts_manager.cc:194] Registered new tserver with Master: ea50a1c39d174993ae777a27876c8403 (127.0.98.3:36755)
I20260928 04:53:57.560513 6496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:53:57.562219 6496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:53:57.571118 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.566457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36755 (local address 127.0.0.1:53812)
0928 04:53:57.566529 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.566543 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.566588 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.567291 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.567292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.567294 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:57.567344 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.567345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.568827 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.568828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.569719 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.569722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.570832 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.570833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.570898 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.570902 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.570963 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.571014 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 04:53:57.571367 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.566564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53812 (local address 127.0.98.3:36755)
0928 04:53:57.567067 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.567071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.567081 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:53:57.567124 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.567126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.567142 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.567297 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:57.567452 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.568719 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.569836 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.570734 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.570823 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.570919 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.570985 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.571036 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":103,"threads_started":1}
I20260928 04:53:57.571808 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.566475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41419 (local address 127.0.0.1:45248)
0928 04:53:57.566881 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.566891 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.566939 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.567276 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.567277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.567279 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:57.567337 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.567338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.568774 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.568776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.569912 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.569915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.571557 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.571558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.571615 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.571619 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.571667 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.571749 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":29,"thread_start_us":105,"threads_started":1}
I20260928 04:53:57.572067 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.566728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45248 (local address 127.0.98.2:41419)
0928 04:53:57.566987 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.566992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.567004 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:53:57.567057 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.567059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.567078 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.567204 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:57.567403 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.568670 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.570618 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.571358 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.571442 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.571519 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.571937 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.571983 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":108,"threads_started":1}
I20260928 04:53:57.572242 7669 tablet_service.cc:1467] Processing CreateTablet for tablet 377bde6bb64742729e6b616dc8fe9e5c (DEFAULT_TABLE table=test-table [id=46de4c2b33e94428b7426e9502762569]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:57.572710 7669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377bde6bb64742729e6b616dc8fe9e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:57.572793 7533 tablet_service.cc:1467] Processing CreateTablet for tablet 377bde6bb64742729e6b616dc8fe9e5c (DEFAULT_TABLE table=test-table [id=46de4c2b33e94428b7426e9502762569]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:57.573119 7533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377bde6bb64742729e6b616dc8fe9e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:57.574761 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.566775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59806 (local address 127.0.98.1:40899)
0928 04:53:57.566985 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.566991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.567103 (+ 112us) server_negotiation.cc:407] Connection header received
0928 04:53:57.567158 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.567160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.567177 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.567240 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:57.567414 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.568399 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.570015 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.570878 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.570952 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.571025 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.574587 (+ 3562us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.574656 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
I20260928 04:53:57.574983 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.566475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40899 (local address 127.0.0.1:59806)
0928 04:53:57.567000 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.567007 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.567047 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.567297 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.567298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.567299 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:57.567336 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.567338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.568503 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.568505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.569912 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.569915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.571065 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.571066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.571137 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.571141 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.571200 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.571268 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":3,"thread_start_us":108,"threads_started":1}
I20260928 04:53:57.576654 7761 tablet_bootstrap.cc:492] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403: Bootstrap starting.
I20260928 04:53:57.577409 7761 tablet_bootstrap.cc:654] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:57.577390 7396 tablet_service.cc:1467] Processing CreateTablet for tablet 377bde6bb64742729e6b616dc8fe9e5c (DEFAULT_TABLE table=test-table [id=46de4c2b33e94428b7426e9502762569]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:53:57.577770 7761 log.cc:824] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:57.577840 7396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377bde6bb64742729e6b616dc8fe9e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:53:57.578724 7762 tablet_bootstrap.cc:492] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da: Bootstrap starting.
I20260928 04:53:57.579319 7761 tablet_bootstrap.cc:492] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403: No bootstrap required, opened a new log
I20260928 04:53:57.579403 7761 ts_tablet_manager.cc:1397] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 04:53:57.579412 7762 tablet_bootstrap.cc:654] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:57.579696 7762 log.cc:824] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:57.580394 7762 tablet_bootstrap.cc:492] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da: No bootstrap required, opened a new log
I20260928 04:53:57.580472 7762 ts_tablet_manager.cc:1397] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:53:57.581694 7761 raft_consensus.cc:348] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea50a1c39d174993ae777a27876c8403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36755 } } peers { permanent_uuid: "db43418eb20a4d5ca92826ab4c6db24e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40899 } } peers { permanent_uuid: "62ddf5590453482a9ada0d96856c81da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41419 } }
I20260928 04:53:57.581918 7761 raft_consensus.cc:374] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:57.581957 7761 raft_consensus.cc:729] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea50a1c39d174993ae777a27876c8403, State: Initialized, Role: FOLLOWER
I20260928 04:53:57.582094 7761 consensus_queue.cc:260] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea50a1c39d174993ae777a27876c8403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36755 } } peers { permanent_uuid: "db43418eb20a4d5ca92826ab4c6db24e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40899 } } peers { permanent_uuid: "62ddf5590453482a9ada0d96856c81da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41419 } }
I20260928 04:53:57.582371 7761 ts_tablet_manager.cc:1428] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:53:57.582523 7736 heartbeater.cc:503] Master 127.0.98.62:44343 was elected leader, sending a full tablet report...
I20260928 04:53:57.582561 7762 raft_consensus.cc:348] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea50a1c39d174993ae777a27876c8403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36755 } } peers { permanent_uuid: "db43418eb20a4d5ca92826ab4c6db24e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40899 } } peers { permanent_uuid: "62ddf5590453482a9ada0d96856c81da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41419 } }
I20260928 04:53:57.582710 7762 raft_consensus.cc:374] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:57.582749 7762 raft_consensus.cc:729] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ddf5590453482a9ada0d96856c81da, State: Initialized, Role: FOLLOWER
I20260928 04:53:57.582876 7762 consensus_queue.cc:260] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea50a1c39d174993ae777a27876c8403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36755 } } peers { permanent_uuid: "db43418eb20a4d5ca92826ab4c6db24e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40899 } } peers { permanent_uuid: "62ddf5590453482a9ada0d96856c81da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41419 } }
I20260928 04:53:57.583189 7762 ts_tablet_manager.cc:1428] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:53:57.583281 7600 heartbeater.cc:503] Master 127.0.98.62:44343 was elected leader, sending a full tablet report...
I20260928 04:53:57.584242 7765 tablet_bootstrap.cc:492] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e: Bootstrap starting.
I20260928 04:53:57.584968 7765 tablet_bootstrap.cc:654] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e: Neither blocks nor log segments found. Creating new log.
I20260928 04:53:57.585239 7765 log.cc:824] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:53:57.586063 7765 tablet_bootstrap.cc:492] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e: No bootstrap required, opened a new log
I20260928 04:53:57.586126 7765 ts_tablet_manager.cc:1397] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:53:57.588340 7765 raft_consensus.cc:348] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea50a1c39d174993ae777a27876c8403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36755 } } peers { permanent_uuid: "db43418eb20a4d5ca92826ab4c6db24e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40899 } } peers { permanent_uuid: "62ddf5590453482a9ada0d96856c81da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41419 } }
I20260928 04:53:57.588519 7765 raft_consensus.cc:374] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:53:57.588544 7765 raft_consensus.cc:729] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db43418eb20a4d5ca92826ab4c6db24e, State: Initialized, Role: FOLLOWER
I20260928 04:53:57.588677 7765 consensus_queue.cc:260] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea50a1c39d174993ae777a27876c8403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36755 } } peers { permanent_uuid: "db43418eb20a4d5ca92826ab4c6db24e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40899 } } peers { permanent_uuid: "62ddf5590453482a9ada0d96856c81da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41419 } }
I20260928 04:53:57.588972 7765 ts_tablet_manager.cc:1428] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:53:57.589033 7463 heartbeater.cc:503] Master 127.0.98.62:44343 was elected leader, sending a full tablet report...
I20260928 04:53:57.630398 7766 raft_consensus.cc:482] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:53:57.630592 7766 raft_consensus.cc:504] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea50a1c39d174993ae777a27876c8403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36755 } } peers { permanent_uuid: "db43418eb20a4d5ca92826ab4c6db24e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40899 } } peers { permanent_uuid: "62ddf5590453482a9ada0d96856c81da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41419 } }
I20260928 04:53:57.631021 7766 leader_election.cc:288] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db43418eb20a4d5ca92826ab4c6db24e (127.0.98.1:40899), 62ddf5590453482a9ada0d96856c81da (127.0.98.2:41419)
I20260928 04:53:57.634227 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.631270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57917 (local address 127.0.98.1:40899)
0928 04:53:57.631318 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.631320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.631331 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:53:57.631421 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.631422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.631446 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.631494 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:57.631713 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.632261 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.633397 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.633813 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.633874 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.633884 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.634085 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.634141 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:53:57.634265 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.631366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52641 (local address 127.0.98.2:41419)
0928 04:53:57.631411 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.631413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.631420 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:53:57.631450 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.631451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.631463 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.631561 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:53:57.631785 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.632352 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.633394 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.633826 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.633929 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.633941 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.634147 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.634197 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:53:57.634262 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.631187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40899 (local address 127.0.98.3:57917)
0928 04:53:57.631256 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.631269 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.631319 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.631532 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.631533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.631535 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:57.631633 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.631635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.632358 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.632360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.633280 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.633282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.633909 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.633911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.634005 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.634008 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.634057 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.634125 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":27}
I20260928 04:53:57.634637 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.631219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41419 (local address 127.0.98.3:52641)
0928 04:53:57.631307 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:57.631318 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:53:57.631356 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:53:57.631574 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:53:57.631575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:53:57.631578 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:53:57.631658 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.631660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.632444 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.632444 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.633283 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:53:57.633285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.633944 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.633945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:53:57.634030 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.634034 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:53:57.634129 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 04:53:57.634173 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 04:53:57.634768 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377bde6bb64742729e6b616dc8fe9e5c" candidate_uuid: "ea50a1c39d174993ae777a27876c8403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db43418eb20a4d5ca92826ab4c6db24e" is_pre_election: true
I20260928 04:53:57.634999 7416 raft_consensus.cc:2454] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea50a1c39d174993ae777a27876c8403 in term 0.
I20260928 04:53:57.635030 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377bde6bb64742729e6b616dc8fe9e5c" candidate_uuid: "ea50a1c39d174993ae777a27876c8403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62ddf5590453482a9ada0d96856c81da" is_pre_election: true
I20260928 04:53:57.635229 7553 raft_consensus.cc:2454] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea50a1c39d174993ae777a27876c8403 in term 0.
I20260928 04:53:57.635391 7625 leader_election.cc:302] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db43418eb20a4d5ca92826ab4c6db24e, ea50a1c39d174993ae777a27876c8403; no voters:
I20260928 04:53:57.635593 7766 raft_consensus.cc:2781] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:53:57.635658 7766 raft_consensus.cc:482] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:53:57.635676 7766 raft_consensus.cc:3037] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:57.636431 7766 raft_consensus.cc:504] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea50a1c39d174993ae777a27876c8403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36755 } } peers { permanent_uuid: "db43418eb20a4d5ca92826ab4c6db24e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40899 } } peers { permanent_uuid: "62ddf5590453482a9ada0d96856c81da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41419 } }
I20260928 04:53:57.636591 7766 leader_election.cc:288] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [CANDIDATE]: Term 1 election: Requested vote from peers db43418eb20a4d5ca92826ab4c6db24e (127.0.98.1:40899), 62ddf5590453482a9ada0d96856c81da (127.0.98.2:41419)
I20260928 04:53:57.636792 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377bde6bb64742729e6b616dc8fe9e5c" candidate_uuid: "ea50a1c39d174993ae777a27876c8403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62ddf5590453482a9ada0d96856c81da"
I20260928 04:53:57.636772 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377bde6bb64742729e6b616dc8fe9e5c" candidate_uuid: "ea50a1c39d174993ae777a27876c8403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db43418eb20a4d5ca92826ab4c6db24e"
I20260928 04:53:57.636865 7553 raft_consensus.cc:3037] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:57.636888 7416 raft_consensus.cc:3037] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:53:57.637612 7553 raft_consensus.cc:2454] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea50a1c39d174993ae777a27876c8403 in term 1.
I20260928 04:53:57.637622 7416 raft_consensus.cc:2454] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea50a1c39d174993ae777a27876c8403 in term 1.
I20260928 04:53:57.637813 7623 leader_election.cc:302] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62ddf5590453482a9ada0d96856c81da, ea50a1c39d174993ae777a27876c8403; no voters:
I20260928 04:53:57.637913 7766 raft_consensus.cc:2781] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:53:57.638087 7766 raft_consensus.cc:686] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [term 1 LEADER]: Becoming Leader. State: Replica: ea50a1c39d174993ae777a27876c8403, State: Running, Role: LEADER
I20260928 04:53:57.638167 7766 consensus_queue.cc:237] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea50a1c39d174993ae777a27876c8403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36755 } } peers { permanent_uuid: "db43418eb20a4d5ca92826ab4c6db24e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40899 } } peers { permanent_uuid: "62ddf5590453482a9ada0d96856c81da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41419 } }
I20260928 04:53:57.639182 6497 catalog_manager.cc:5495] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea50a1c39d174993ae777a27876c8403 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ea50a1c39d174993ae777a27876c8403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea50a1c39d174993ae777a27876c8403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db43418eb20a4d5ca92826ab4c6db24e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62ddf5590453482a9ada0d96856c81da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41419 } health_report { overall_health: UNKNOWN } } }
W20260928 04:53:57.644414 7602 tablet.cc:2369] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:53:57.645577 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.642819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53820 (local address 127.0.98.3:36755)
0928 04:53:57.642875 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.642877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.642885 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:53:57.642960 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.642962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.642982 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.643042 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:53:57.643208 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.643676 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.644662 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.644915 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.645051 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.645264 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:53:57.645376 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:53:57.645456 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.645492 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.645516 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 04:53:57.647787 7553 raft_consensus.cc:1261] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da [term 1 FOLLOWER]: Refusing update from remote peer ea50a1c39d174993ae777a27876c8403: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:57.647781 7416 raft_consensus.cc:1261] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e [term 1 FOLLOWER]: Refusing update from remote peer ea50a1c39d174993ae777a27876c8403: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:53:57.648203 7770 consensus_queue.cc:1035] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ddf5590453482a9ada0d96856c81da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:57.648356 7766 consensus_queue.cc:1035] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db43418eb20a4d5ca92826ab4c6db24e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:53:57.651824 7776 mvcc.cc:205] Tried to move back new op lower bound from 7334179789402271744 to 7334179789367107584. Current Snapshot: MvccSnapshot[applied={T|T < 7334179789402271744}]
I20260928 04:53:57.655565 6496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:53:57.655747 6496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:53:57.657315 6496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55fc8a9c3dd54c1eaf1dece4e8993857: Sending DeleteTablet for 3 replicas of tablet 377bde6bb64742729e6b616dc8fe9e5c
I20260928 04:53:57.657675 7669 tablet_service.cc:1514] Processing DeleteTablet for tablet 377bde6bb64742729e6b616dc8fe9e5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53812
I20260928 04:53:57.657766 7396 tablet_service.cc:1514] Processing DeleteTablet for tablet 377bde6bb64742729e6b616dc8fe9e5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59806
I20260928 04:53:57.658013 7785 tablet_replica.cc:330] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e: stopping tablet replica
I20260928 04:53:57.658138 7785 raft_consensus.cc:2229] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:53:57.658286 7785 raft_consensus.cc:2258] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:57.658455 7784 tablet_replica.cc:330] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403: stopping tablet replica
I20260928 04:53:57.658589 7784 raft_consensus.cc:2229] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:53:57.658726 7784 raft_consensus.cc:2258] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:53:57.658710 7533 tablet_service.cc:1514] Processing DeleteTablet for tablet 377bde6bb64742729e6b616dc8fe9e5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:53:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45248
I20260928 04:53:57.658954 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 6465
I20260928 04:53:57.659260 7784 ts_tablet_manager.cc:1905] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:57.660980 7784 ts_tablet_manager.cc:1918] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:53:57.661058 7784 log.cc:1196] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/377bde6bb64742729e6b616dc8fe9e5c
I20260928 04:53:57.661336 7784 ts_tablet_manager.cc:1939] T 377bde6bb64742729e6b616dc8fe9e5c P ea50a1c39d174993ae777a27876c8403: Deleting consensus metadata
I20260928 04:53:57.661839 7785 ts_tablet_manager.cc:1905] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:57.663931 7785 ts_tablet_manager.cc:1918] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:53:57.664027 7785 log.cc:1196] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-0/wal/wals/377bde6bb64742729e6b616dc8fe9e5c
I20260928 04:53:57.664325 7785 ts_tablet_manager.cc:1939] T 377bde6bb64742729e6b616dc8fe9e5c P db43418eb20a4d5ca92826ab4c6db24e: Deleting consensus metadata
I20260928 04:53:57.667052 7787 tablet_replica.cc:330] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da: stopping tablet replica
W20260928 04:53:57.667444 7352 connection.cc:535] server connection from 127.0.0.1:59806 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 04:53:57.667896 7787 raft_consensus.cc:2229] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 04:53:57.667877 7623 connection.cc:441] server connection from 127.0.0.1:53812 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53812 (request call id 1) could send its response
I20260928 04:53:57.668159 7787 raft_consensus.cc:2258] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 04:53:57.668546 7082 connection.cc:535] server connection from 127.0.0.1:47772 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 04:53:57.668838 7787 ts_tablet_manager.cc:1905] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:53:57.669691 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.658396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.0.98.60:43627)
0928 04:53:57.658445 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.658447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.659567 (+ 1120us) server_negotiation.cc:407] Connection header received
0928 04:53:57.659679 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.659680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.659705 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.659740 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.660759 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.661663 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.663203 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.663577 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.663623 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.666347 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.666349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.666349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.666359 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.668177 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.668813 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.668815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.668815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.668877 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.669074 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.669075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.669075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.669223 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.669299 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.669510 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.669594 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:53:57.671429 7787 ts_tablet_manager.cc:1918] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:53:57.671531 7787 log.cc:1196] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/377bde6bb64742729e6b616dc8fe9e5c
I20260928 04:53:57.671959 7787 ts_tablet_manager.cc:1939] T 377bde6bb64742729e6b616dc8fe9e5c P 62ddf5590453482a9ada0d96856c81da: Deleting consensus metadata
W20260928 04:53:57.672428 7488 connection.cc:441] server connection from 127.0.0.1:45248 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45248 (request call id 1) could send its response
I20260928 04:53:57.679580 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.670692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40288 (local address 127.0.98.60:43627)
0928 04:53:57.670726 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.670728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.671115 (+ 387us) server_negotiation.cc:407] Connection header received
0928 04:53:57.671278 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.671279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.671292 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.671378 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.672438 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.673102 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.674028 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.674261 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.674361 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.676061 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.676063 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.676063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.676071 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.678045 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.678551 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.678552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.678553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.678620 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.678911 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.678912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.678913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.679059 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.679133 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.679376 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.679471 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 04:53:57.679600 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:57.670454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47788 (local address 127.0.98.61:43647)
0928 04:53:57.670530 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 04:53:57.670532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:53:57.671058 (+ 526us) server_negotiation.cc:407] Connection header received
0928 04:53:57.671167 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:53:57.671168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:53:57.671180 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:53:57.671211 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:53:57.671977 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.672834 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.673937 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:53:57.674265 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:53:57.674308 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:53:57.676131 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:53:57.676133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:53:57.676133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:53:57.676140 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:53:57.678239 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.678739 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.678741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.678741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.678809 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:53:57.679048 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:53:57.679049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:53:57.679049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:53:57.679173 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:53:57.679284 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 04:53:57.679433 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 04:53:57.679507 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:53:58.653328 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:58.652981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:39015)
0928 04:53:58.653181 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:58.653272 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":135,"threads_started":1}
W20260928 04:53:58.653750 7600 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44343 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
I20260928 04:53:58.653777 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:58.653392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:41711)
0928 04:53:58.653647 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:58.653722 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":58,"threads_started":1}
I20260928 04:53:58.653935 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:58.653718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:56253)
0928 04:53:58.653832 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:58.653894 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
W20260928 04:53:58.653942 7463 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44343 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
W20260928 04:53:58.654160 7736 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44343 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
I20260928 04:53:59.157429 7798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 55fc8a9c3dd54c1eaf1dece4e8993857)
I20260928 04:53:59.157583 7798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: false } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } }
I20260928 04:53:59.157867 7798 leader_election.cc:288] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55fc8a9c3dd54c1eaf1dece4e8993857 (127.0.98.62:44343)
I20260928 04:53:59.158264 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:59.158057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.0.1:34240)
0928 04:53:59.158169 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:59.158227 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
W20260928 04:53:59.158401 7081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
I20260928 04:53:59.158867 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:59.158760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.0.1:34242)
0928 04:53:59.158811 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:59.158840 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 04:53:59.158962 7081 leader_election.cc:334] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 55fc8a9c3dd54c1eaf1dece4e8993857 (127.0.98.62:44343): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
I20260928 04:53:59.159004 7081 leader_election.cc:302] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 065816551ea64988a938a5e2e163db38; no voters: 55fc8a9c3dd54c1eaf1dece4e8993857
I20260928 04:53:59.159096 7798 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:53:59.654335 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:59.654071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:43201)
0928 04:53:59.654218 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:59.654269 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260928 04:53:59.654434 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:59.654202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:59945)
0928 04:53:59.654345 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:59.654382 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260928 04:53:59.654768 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:53:59.654419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:36267)
0928 04:53:59.654691 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 04:53:59.654727 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":52,"threads_started":1}
I20260928 04:54:00.655385 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:00.655087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:50521)
0928 04:54:00.655293 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:00.655334 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":76,"threads_started":1}
I20260928 04:54:00.655385 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:00.655177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:37287)
0928 04:54:00.655303 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:00.655334 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":46,"threads_started":1}
I20260928 04:54:00.655416 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:00.655210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:52603)
0928 04:54:00.655329 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:00.655364 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260928 04:54:00.676023 7807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 55fc8a9c3dd54c1eaf1dece4e8993857)
I20260928 04:54:00.676136 7807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: false } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } }
I20260928 04:54:00.676342 7807 leader_election.cc:288] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55fc8a9c3dd54c1eaf1dece4e8993857 (127.0.98.62:44343)
I20260928 04:54:00.676779 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:00.676543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.0.1:34440)
0928 04:54:00.676704 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:00.676741 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
I20260928 04:54:00.677227 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:00.677098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.0.1:34444)
0928 04:54:00.677169 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:00.677196 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 04:54:00.677307 7081 leader_election.cc:334] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 55fc8a9c3dd54c1eaf1dece4e8993857 (127.0.98.62:44343): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
I20260928 04:54:00.677345 7081 leader_election.cc:302] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 065816551ea64988a938a5e2e163db38; no voters: 55fc8a9c3dd54c1eaf1dece4e8993857
I20260928 04:54:00.677412 7807 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:54:01.656177 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:01.655928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:41231)
0928 04:54:01.656066 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:01.656105 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260928 04:54:01.656159 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:01.655906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:40591)
0928 04:54:01.656066 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:01.656105 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260928 04:54:01.656265 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:01.655928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:40709)
0928 04:54:01.656165 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:01.656209 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":67,"threads_started":1}
I20260928 04:54:02.656962 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:02.656728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:44203)
0928 04:54:02.656859 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:02.656912 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":53,"threads_started":1}
I20260928 04:54:02.656958 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:02.656732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:43771)
0928 04:54:02.656854 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:02.656898 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260928 04:54:02.657078 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:02.656708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:53563)
0928 04:54:02.656999 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:02.657036 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":70,"threads_started":1}
I20260928 04:54:02.993018 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 55fc8a9c3dd54c1eaf1dece4e8993857)
I20260928 04:54:02.993130 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: false } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } }
I20260928 04:54:02.993299 7816 leader_election.cc:288] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55fc8a9c3dd54c1eaf1dece4e8993857 (127.0.98.62:44343)
I20260928 04:54:02.993706 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:02.993469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.0.1:34708)
0928 04:54:02.993621 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:02.993666 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":48,"threads_started":1}
I20260928 04:54:02.994205 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:02.994098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.0.1:34710)
0928 04:54:02.994155 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:02.994185 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 04:54:02.994339 7081 leader_election.cc:334] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 55fc8a9c3dd54c1eaf1dece4e8993857 (127.0.98.62:44343): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
I20260928 04:54:02.994423 7081 leader_election.cc:302] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 065816551ea64988a938a5e2e163db38; no voters: 55fc8a9c3dd54c1eaf1dece4e8993857
I20260928 04:54:02.994566 7816 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:54:03.657738 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:03.657451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:59369)
0928 04:54:03.657627 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:03.657682 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":81,"threads_started":1}
I20260928 04:54:03.657737 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:03.657451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:59699)
0928 04:54:03.657623 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:03.657672 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":70,"threads_started":1}
I20260928 04:54:03.657820 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:03.657644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:52755)
0928 04:54:03.657746 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:03.657777 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":51,"threads_started":1}
I20260928 04:54:04.658627 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:04.658313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:38515)
0928 04:54:04.658442 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:04.658536 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
I20260928 04:54:04.658854 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:04.658291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:56291)
0928 04:54:04.658747 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:04.658791 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":78,"threads_started":1}
I20260928 04:54:04.658892 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:04.658708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:50477)
0928 04:54:04.658820 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:04.658849 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260928 04:54:05.659570 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:05.659206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:44715)
0928 04:54:05.659464 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:05.659505 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":70,"threads_started":1}
I20260928 04:54:05.659570 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:05.659332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:49565)
0928 04:54:05.659461 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:05.659526 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260928 04:54:05.659603 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:05.659294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:41555)
0928 04:54:05.659419 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:05.659462 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":54,"threads_started":1}
I20260928 04:54:06.660353 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:06.660089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:33909)
0928 04:54:06.660241 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:06.660287 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":57,"threads_started":1}
I20260928 04:54:06.660640 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:06.660074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:56459)
0928 04:54:06.660539 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:06.660589 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":83,"threads_started":1}
I20260928 04:54:06.660655 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:06.660256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:47947)
0928 04:54:06.660561 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:06.660609 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":48,"threads_started":1}
I20260928 04:54:07.473829 7831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 55fc8a9c3dd54c1eaf1dece4e8993857)
I20260928 04:54:07.473937 7831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: false } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } }
I20260928 04:54:07.474161 7831 leader_election.cc:288] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55fc8a9c3dd54c1eaf1dece4e8993857 (127.0.98.62:44343)
I20260928 04:54:07.474619 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:07.474377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.0.1:34074)
0928 04:54:07.474528 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:07.474577 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":70,"threads_started":1}
W20260928 04:54:07.474713 7081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260928 04:54:07.475124 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:07.475001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.0.1:34088)
0928 04:54:07.475059 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:07.475100 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 04:54:07.475255 7081 leader_election.cc:334] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 55fc8a9c3dd54c1eaf1dece4e8993857 (127.0.98.62:44343): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
I20260928 04:54:07.475329 7081 leader_election.cc:302] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 065816551ea64988a938a5e2e163db38; no voters: 55fc8a9c3dd54c1eaf1dece4e8993857
I20260928 04:54:07.475458 7831 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:54:07.661329 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:07.661086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:60685)
0928 04:54:07.661230 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:07.661269 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260928 04:54:07.661324 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:07.661086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:52893)
0928 04:54:07.661230 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:07.661268 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260928 04:54:07.661379 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:07.661123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:37765)
0928 04:54:07.661287 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:07.661326 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
I20260928 04:54:08.661926 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:08.661693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:43547)
0928 04:54:08.661827 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:08.661870 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260928 04:54:08.661983 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:08.661712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:58781)
0928 04:54:08.661839 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:08.661883 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260928 04:54:08.662083 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:08.661759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:56081)
0928 04:54:08.661999 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:08.662036 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":47,"threads_started":1}
I20260928 04:54:09.662745 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:09.662503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:50441)
0928 04:54:09.662651 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:09.662695 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260928 04:54:09.662745 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:09.662530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:59633)
0928 04:54:09.662665 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:09.662694 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260928 04:54:09.662782 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:09.662572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:57139)
0928 04:54:09.662688 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:09.662727 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":46,"threads_started":1}
I20260928 04:54:10.663393 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:10.663178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:40347)
0928 04:54:10.663304 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:10.663340 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260928 04:54:10.663393 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:10.663177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:54001)
0928 04:54:10.663303 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:10.663340 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260928 04:54:10.663564 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:10.663345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:58925)
0928 04:54:10.663478 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:10.663513 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":55,"threads_started":1}
I20260928 04:54:11.664016 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:11.663795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:43547)
0928 04:54:11.663920 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:11.663968 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260928 04:54:11.664124 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:11.663930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:44497)
0928 04:54:11.664025 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:11.664059 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":42,"threads_started":1}
I20260928 04:54:11.664186 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:11.663872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:49153)
0928 04:54:11.664108 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:11.664141 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":59,"threads_started":1}
I20260928 04:54:12.664880 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:12.664620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:55921)
0928 04:54:12.664775 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:12.664819 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260928 04:54:12.664885 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:12.664612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:46017)
0928 04:54:12.664791 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:12.664831 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":66,"threads_started":1}
I20260928 04:54:12.664963 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:12.664801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:47845)
0928 04:54:12.664895 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:12.664922 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":45,"threads_started":1}
I20260928 04:54:13.665598 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:13.665301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:36411)
0928 04:54:13.665478 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:13.665521 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
I20260928 04:54:13.665720 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:13.665504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:36055)
0928 04:54:13.665641 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:13.665676 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
I20260928 04:54:13.665844 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:13.665296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:59673)
0928 04:54:13.665743 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:13.665790 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
I20260928 04:54:14.624789 7855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 55fc8a9c3dd54c1eaf1dece4e8993857)
I20260928 04:54:14.624898 7855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: false } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } }
I20260928 04:54:14.625128 7855 leader_election.cc:288] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55fc8a9c3dd54c1eaf1dece4e8993857 (127.0.98.62:44343)
I20260928 04:54:14.625543 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:14.625334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.0.1:45274)
0928 04:54:14.625459 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:14.625500 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":85,"threads_started":1}
W20260928 04:54:14.625639 7081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
I20260928 04:54:14.626051 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:14.625962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.0.1:45276)
0928 04:54:14.626007 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:14.626033 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 04:54:14.626137 7081 leader_election.cc:334] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 55fc8a9c3dd54c1eaf1dece4e8993857 (127.0.98.62:44343): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
I20260928 04:54:14.626214 7081 leader_election.cc:302] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 065816551ea64988a938a5e2e163db38; no voters: 55fc8a9c3dd54c1eaf1dece4e8993857
I20260928 04:54:14.626284 7855 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:54:14.666440 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:14.666096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:55195)
0928 04:54:14.666352 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:14.666389 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":82,"threads_started":1}
I20260928 04:54:14.666440 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:14.666237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:33203)
0928 04:54:14.666348 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:14.666388 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":49,"threads_started":1}
I20260928 04:54:14.666582 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:14.666210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:43667)
0928 04:54:14.666457 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:14.666527 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":78,"threads_started":1}
I20260928 04:54:15.667129 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:15.666862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:33805)
0928 04:54:15.667017 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:15.667065 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260928 04:54:15.667145 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:15.666847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:50051)
0928 04:54:15.667049 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:15.667088 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
I20260928 04:54:15.667311 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:15.667062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:40279)
0928 04:54:15.667188 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:15.667214 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260928 04:54:16.667905 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:16.667672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:42565)
0928 04:54:16.667801 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:16.667846 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":54,"threads_started":1}
I20260928 04:54:16.667961 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:16.667532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:38971)
0928 04:54:16.667886 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:16.667920 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":111,"threads_started":1}
I20260928 04:54:16.667966 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:16.667709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:55595)
0928 04:54:16.667877 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:16.667915 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":45,"threads_started":1}
I20260928 04:54:17.668726 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:17.668530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:59535)
0928 04:54:17.668659 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:17.668688 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260928 04:54:17.668726 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:17.668427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:41227)
0928 04:54:17.668617 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:17.668668 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260928 04:54:17.669018 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:17.668562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:46875)
0928 04:54:17.668921 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:17.668962 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":48,"threads_started":1}
I20260928 04:54:18.669540 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:18.669194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:38181)
0928 04:54:18.669433 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:18.669477 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":86,"threads_started":1}
I20260928 04:54:18.669610 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:18.669152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:40489)
0928 04:54:18.669518 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:18.669557 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":70,"threads_started":1}
I20260928 04:54:18.669654 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:18.669351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:52271)
0928 04:54:18.669582 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:18.669610 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260928 04:54:19.670311 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:19.670043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:39671)
0928 04:54:19.670209 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:19.670251 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":70,"threads_started":1}
I20260928 04:54:19.670320 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:19.670044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:46775)
0928 04:54:19.670225 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:19.670264 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
I20260928 04:54:19.670647 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:19.670399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:57949)
0928 04:54:19.670571 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:19.670613 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":50,"threads_started":1}
I20260928 04:54:20.671082 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:20.670822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:51061)
0928 04:54:20.670966 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:20.671020 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260928 04:54:20.671197 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:20.670916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:58461)
0928 04:54:20.671063 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:20.671144 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":45,"threads_started":1}
I20260928 04:54:20.671352 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:20.671070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:41763)
0928 04:54:20.671272 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:20.671310 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":51,"threads_started":1}
I20260928 04:54:21.672034 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:21.671649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:50425)
0928 04:54:21.671937 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:21.671977 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":60,"threads_started":1}
I20260928 04:54:21.672034 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:21.671640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:60933)
0928 04:54:21.671934 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:21.671978 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":64,"threads_started":1}
I20260928 04:54:21.672264 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:21.671833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:37539)
0928 04:54:21.671980 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:21.672022 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":46,"threads_started":1}
I20260928 04:54:22.672889 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:22.672596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:47149)
0928 04:54:22.672755 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:22.672796 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260928 04:54:22.672974 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:22.672804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:47299)
0928 04:54:22.672898 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:22.672929 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":46,"threads_started":1}
I20260928 04:54:22.672974 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:22.672783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:41699)
0928 04:54:22.672898 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:22.672929 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260928 04:54:23.673866 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:23.673520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:33291)
0928 04:54:23.673779 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:23.673809 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":72,"threads_started":1}
I20260928 04:54:23.673874 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:23.673520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:47589)
0928 04:54:23.673759 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:23.673807 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":69,"threads_started":1}
I20260928 04:54:23.674026 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:23.673559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:57109)
0928 04:54:23.673838 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:23.673971 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":55,"threads_started":1}
I20260928 04:54:24.674710 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:24.674429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:54981)
0928 04:54:24.674605 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:24.674651 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":91,"threads_started":1}
I20260928 04:54:24.674826 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:24.674465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:43095)
0928 04:54:24.674739 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:24.674780 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":67,"threads_started":1}
I20260928 04:54:24.674834 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:24.674665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:60253)
0928 04:54:24.674773 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:24.674798 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
I20260928 04:54:25.675609 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:25.675232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:60927)
0928 04:54:25.675491 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:25.675539 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":73,"threads_started":1}
I20260928 04:54:25.675665 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:25.675198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:45375)
0928 04:54:25.675585 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:25.675619 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":68,"threads_started":1}
I20260928 04:54:25.675810 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:25.675216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:37395)
0928 04:54:25.675647 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:25.675705 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":53,"threads_started":1}
I20260928 04:54:25.976794 7894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 55fc8a9c3dd54c1eaf1dece4e8993857)
I20260928 04:54:25.976899 7894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55fc8a9c3dd54c1eaf1dece4e8993857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44343 } } peers { permanent_uuid: "065816551ea64988a938a5e2e163db38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attrs { promote: false } } peers { permanent_uuid: "920bfb0d0bb64d6baf003fbe1d5a99ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43627 } attrs { promote: true } }
I20260928 04:54:25.977033 7894 leader_election.cc:288] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55fc8a9c3dd54c1eaf1dece4e8993857 (127.0.98.62:44343)
I20260928 04:54:25.977471 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:25.977212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.0.1:54344)
0928 04:54:25.977342 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:25.977386 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
W20260928 04:54:25.977566 7081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
I20260928 04:54:25.977952 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:25.977851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.0.1:54350)
0928 04:54:25.977905 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:25.977933 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 04:54:25.978039 7081 leader_election.cc:334] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 55fc8a9c3dd54c1eaf1dece4e8993857 (127.0.98.62:44343): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
I20260928 04:54:25.978098 7081 leader_election.cc:302] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 065816551ea64988a938a5e2e163db38; no voters: 55fc8a9c3dd54c1eaf1dece4e8993857
I20260928 04:54:25.978189 7894 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 065816551ea64988a938a5e2e163db38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:54:26.676530 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:26.676252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:47625)
0928 04:54:26.676411 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:26.676458 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
I20260928 04:54:26.676574 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:26.676336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:56205)
0928 04:54:26.676474 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:26.676514 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260928 04:54:26.676659 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:26.676465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:44075)
0928 04:54:26.676584 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:26.676614 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260928 04:54:27.677362 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:27.677120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.2:49045)
0928 04:54:27.677226 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:27.677272 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":56,"threads_started":1}
I20260928 04:54:27.677407 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:27.677080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.1:52141)
0928 04:54:27.677209 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:27.677350 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
I20260928 04:54:27.677556 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:27.677308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44343 (local address 127.0.98.3:33729)
0928 04:54:27.677425 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:27.677462 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.98.62:44343,127.0.98.61:43647,127.0.98.60:43627, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.98.62:44343,127.0.98.61:43647,127.0.98.60:43627, num_attempts: 1)
I20260928 04:54:27.895061 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 7244
I20260928 04:54:27.900473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 7470
I20260928 04:54:27.905473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 7606
I20260928 04:54:27.910883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 6962
I20260928 04:54:27.915966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 7150
2026-09-28T04:54:27Z chronyd exiting
I20260928 04:54:27.932564 392 test_util.cc:175] -----------------------------------------------
I20260928 04:54:27.932669 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790571208412331-392-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33276 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:54:27 dist-test-slave-m39l krb5kdc[7905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:54:27 dist-test-slave-m39l krb5kdc[7905](info): set up 2 sockets
Sep 28 04:54:27 dist-test-slave-m39l krb5kdc[7905](info): commencing operation
krb5kdc: starting...
W20260928 04:54:28.971011 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:54:29 dist-test-slave-m39l krb5kdc[7905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:54:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:54:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 04:54:29.098341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45175
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:54:29.201112 7918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:29.201341 7918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:29.201366 7918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:29.202811 7918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:54:29.202863 7918 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 04:54:29.202877 7918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:29.202890 7918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:29.202903 7918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:54:29.202915 7918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:54:29.204487 7918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45175
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:29.204769 7918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:29.205042 7918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:29.210260 7924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:29.210249 7923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:29.210409 7918 server_base.cc:1034] running on GCE node
W20260928 04:54:29.210246 7926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:29.210872 7918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:29.211194 7918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:29.212389 7918 hybrid_clock.cc:648] HybridClock initialized: now 1790571269212369 us; error 95 us; skew 500 ppm
Sep 28 04:54:29 dist-test-slave-m39l krb5kdc[7905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:54:29.215169 7918 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 04:54:29.215655 7918 webserver.cc:460] Webserver started at http://127.0.98.62:43219/ using document root <none> and password file <none>
I20260928 04:54:29.215912 7918 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:29.215982 7918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:29.216143 7918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:29.217265 7918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "5821ada73b2747f98afff71a9c0a9e78"
format_stamp: "Formatted at 2026-09-28 04:54:29 on dist-test-slave-m39l"
server_key: "5c284cdcdb584a522a5ccacd29f47c80"
server_key_iv: "15747e8f53f41e1e7e8fbe3868fa397e"
server_key_version: "encryptionkey@0"
I20260928 04:54:29.217643 7918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "5821ada73b2747f98afff71a9c0a9e78"
format_stamp: "Formatted at 2026-09-28 04:54:29 on dist-test-slave-m39l"
server_key: "5c284cdcdb584a522a5ccacd29f47c80"
server_key_iv: "15747e8f53f41e1e7e8fbe3868fa397e"
server_key_version: "encryptionkey@0"
I20260928 04:54:29.219115 7918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:54:29.220033 7933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:29.220263 7918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:54:29.220410 7918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "5821ada73b2747f98afff71a9c0a9e78"
format_stamp: "Formatted at 2026-09-28 04:54:29 on dist-test-slave-m39l"
server_key: "5c284cdcdb584a522a5ccacd29f47c80"
server_key_iv: "15747e8f53f41e1e7e8fbe3868fa397e"
server_key_version: "encryptionkey@0"
I20260928 04:54:29.220500 7918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:29.245165 7918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:29.247695 7918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:29.247874 7918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:29.252692 7918 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45175
I20260928 04:54:29.252785 7995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45175 every 8 connection(s)
I20260928 04:54:29.253131 7918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:54:29.253903 7996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:29.253911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 7918
I20260928 04:54:29.254043 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:54:29.254233 392 external_mini_cluster.cc:1428] Setting key 760266f6f17260780076e0e703de56aa
I20260928 04:54:29.257041 7996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78: Bootstrap starting.
I20260928 04:54:29.257701 7996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:29.258000 7996 log.cc:824] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78: Log is configured to *not* fsync() on all Append() calls
I20260928 04:54:29.258747 7996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78: No bootstrap required, opened a new log
Sep 28 04:54:29 dist-test-slave-m39l krb5kdc[7905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:54:29.260910 7996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5821ada73b2747f98afff71a9c0a9e78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } }
I20260928 04:54:29.261125 7996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:29.261198 7996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5821ada73b2747f98afff71a9c0a9e78, State: Initialized, Role: FOLLOWER
I20260928 04:54:29.261348 7996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5821ada73b2747f98afff71a9c0a9e78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } }
I20260928 04:54:29.261447 7996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:54:29.261494 7996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:54:29.261535 7996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:29.262164 7996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5821ada73b2747f98afff71a9c0a9e78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } }
I20260928 04:54:29.262250 7996 leader_election.cc:302] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5821ada73b2747f98afff71a9c0a9e78; no voters:
I20260928 04:54:29.262480 7996 leader_election.cc:288] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:54:29.262543 8001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:54:29.262681 8001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78 [term 1 LEADER]: Becoming Leader. State: Replica: 5821ada73b2747f98afff71a9c0a9e78, State: Running, Role: LEADER
I20260928 04:54:29.262823 7996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:54:29.262811 8001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5821ada73b2747f98afff71a9c0a9e78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } }
I20260928 04:54:29.263250 8003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5821ada73b2747f98afff71a9c0a9e78. Latest consensus state: current_term: 1 leader_uuid: "5821ada73b2747f98afff71a9c0a9e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5821ada73b2747f98afff71a9c0a9e78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } } }
I20260928 04:54:29.263336 8003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78 [sys.catalog]: This master's current role is: LEADER
I20260928 04:54:29.263605 8010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:54:29.263873 8002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5821ada73b2747f98afff71a9c0a9e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5821ada73b2747f98afff71a9c0a9e78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } } }
I20260928 04:54:29.264026 8002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78 [sys.catalog]: This master's current role is: LEADER
I20260928 04:54:29.264143 8010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:54:29.265533 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.254856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.0.98.62:45175)
0928 04:54:29.255040 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:29.255048 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:29.255075 (+ 27us) server_negotiation.cc:407] Connection header received
0928 04:54:29.255510 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:29.255517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:29.255565 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:29.255720 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:29.256362 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.257347 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.258235 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.258749 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.258865 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.261552 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:29.261558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:29.261563 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:29.261592 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:29.263749 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:29.264200 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:29.264203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:29.264203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:29.264318 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:29.264648 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:29.264649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:29.264649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:29.264910 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:29.265074 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:29.265172 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 04:54:29.265310 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260928 04:54:29.265858 8010 catalog_manager.cc:1347] Generated new cluster ID: 8e7a4341591e452badb6af901f300119
I20260928 04:54:29.265918 8010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:54:29.270089 8010 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:54:29.270870 8010 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:54:29.283628 8010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78: Generated new TSK 0
I20260928 04:54:29.283886 8010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 04:54:29.321775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45175
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:54:29.409013 8022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:29.409271 8022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:29.409324 8022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:29.410674 8022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:29.410743 8022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:29.410776 8022 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:54:29.410818 8022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:54:29.412452 8022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:29.412766 8022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:29.413025 8022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:29.418532 8028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:29.418640 8022 server_base.cc:1034] running on GCE node
W20260928 04:54:29.418640 8030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:29.418527 8027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:29.419166 8022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:29.419482 8022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:29.420665 8022 hybrid_clock.cc:648] HybridClock initialized: now 1790571269420643 us; error 42 us; skew 500 ppm
Sep 28 04:54:29 dist-test-slave-m39l krb5kdc[7905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:54:29.423785 8022 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 04:54:29.424294 8022 webserver.cc:460] Webserver started at http://127.0.98.1:45853/ using document root <none> and password file <none>
I20260928 04:54:29.424599 8022 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:29.424652 8022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:29.424801 8022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:29.425890 8022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "59f9ec8981c54a28b7ea086466a70650"
format_stamp: "Formatted at 2026-09-28 04:54:29 on dist-test-slave-m39l"
server_key: "bd52c70aa194cb0403b8bf229d6371d3"
server_key_iv: "0f5e99cec950edc880bca62c59386d0c"
server_key_version: "encryptionkey@0"
I20260928 04:54:29.426193 8022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "59f9ec8981c54a28b7ea086466a70650"
format_stamp: "Formatted at 2026-09-28 04:54:29 on dist-test-slave-m39l"
server_key: "bd52c70aa194cb0403b8bf229d6371d3"
server_key_iv: "0f5e99cec950edc880bca62c59386d0c"
server_key_version: "encryptionkey@0"
I20260928 04:54:29.427677 8022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:54:29.428556 8037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:29.428802 8022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:54:29.428936 8022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "59f9ec8981c54a28b7ea086466a70650"
format_stamp: "Formatted at 2026-09-28 04:54:29 on dist-test-slave-m39l"
server_key: "bd52c70aa194cb0403b8bf229d6371d3"
server_key_iv: "0f5e99cec950edc880bca62c59386d0c"
server_key_version: "encryptionkey@0"
I20260928 04:54:29.429031 8022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:29.442678 8022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:29.445268 8022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:29.445515 8022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:29.446069 8022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:54:29.446132 8022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:29.446173 8022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:54:29.446198 8022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:29.452917 8022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40913
I20260928 04:54:29.453006 8152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40913 every 8 connection(s)
I20260928 04:54:29.453413 8022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:54:29 dist-test-slave-m39l krb5kdc[7905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:54:29.456959 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.446796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47297 (local address 127.0.98.62:45175)
0928 04:54:29.446863 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:29.446865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:29.447115 (+ 250us) server_negotiation.cc:407] Connection header received
0928 04:54:29.447814 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:29.447815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:29.447828 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:29.447901 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:29.448781 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.449365 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.450932 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.451152 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.451241 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.454080 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:29.454081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:29.454082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:29.454088 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:29.455537 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:29.455960 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:29.455962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:29.455962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:29.456020 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:29.456298 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:29.456298 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:29.456299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:29.456410 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:29.456462 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:29.456798 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 04:54:29.456891 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:54:29.457208 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.446821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45175 (local address 127.0.98.1:47297)
0928 04:54:29.446988 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:29.447024 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:29.447691 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:29.447980 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:29.447985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:29.448372 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:29.448681 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.448687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.449456 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.449458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.450841 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.450844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.451252 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.451253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.451394 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.452007 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:29.452019 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:29.453991 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:29.455616 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:29.455622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:29.455624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:29.455870 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:29.456115 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:29.456116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:29.456117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:29.456231 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:29.456498 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:29.456506 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:29.456663 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 04:54:29.456772 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 04:54:29.456926 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260928 04:54:29.457885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 8022
I20260928 04:54:29.457998 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:54:29.458245 392 external_mini_cluster.cc:1428] Setting key 9778ed208bbee12e29929508b7495bf9
I20260928 04:54:29.458437 8153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45175
I20260928 04:54:29.458593 8153 heartbeater.cc:464] Registering TS with master...
I20260928 04:54:29.458827 8153 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
I20260928 04:54:29.459398 7949 ts_manager.cc:194] Registered new tserver with Master: 59f9ec8981c54a28b7ea086466a70650 (127.0.98.1:40913)
I20260928 04:54:29.460186 7949 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47297
I20260928 04:54:29.465977 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.459159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36785 (local address 127.0.98.62:45175)
0928 04:54:29.459203 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:29.459205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:29.459347 (+ 142us) server_negotiation.cc:407] Connection header received
0928 04:54:29.459382 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:29.459383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:29.459395 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:29.459474 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:29.459847 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.460613 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.461741 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.462125 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.462210 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.463204 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:29.463206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:29.463206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:29.463213 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:29.464785 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:29.465171 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:29.465172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:29.465172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:29.465234 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:29.465434 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:29.465435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:29.465436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:29.465557 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:29.465633 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:29.465790 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 04:54:29.465876 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:54:29.465983 8159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.459124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45175 (local address 127.0.98.1:36785)
0928 04:54:29.459266 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:29.459278 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:29.459338 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:29.459481 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:29.459482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:29.459679 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:29.459760 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.459762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.460715 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.460716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.461632 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.461635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.462213 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.462213 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.462268 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.462803 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:29.462807 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:29.463110 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:29.464883 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:29.464885 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:29.464885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:29.465077 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:29.465300 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:29.465301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:29.465301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:29.465362 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:29.465623 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:29.465624 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:29.465673 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 04:54:29.465761 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 04:54:29.465878 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 04:54:29.490581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45175
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:54:29.576370 8162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:29.576663 8162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:29.576692 8162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:29.577919 8162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:29.577979 8162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:29.578003 8162 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:54:29.578022 8162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:54:29.579648 8162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:29.579900 8162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:29.580098 8162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:29.585803 8168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:29.585803 8167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:29.585819 8170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:29.586093 8162 server_base.cc:1034] running on GCE node
I20260928 04:54:29.586328 8162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:29.586685 8162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:29.587867 8162 hybrid_clock.cc:648] HybridClock initialized: now 1790571269587848 us; error 28 us; skew 500 ppm
Sep 28 04:54:29 dist-test-slave-m39l krb5kdc[7905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:54:29.590649 8162 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 04:54:29.591158 8162 webserver.cc:460] Webserver started at http://127.0.98.2:39451/ using document root <none> and password file <none>
I20260928 04:54:29.591419 8162 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:29.591496 8162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:29.591662 8162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:29.592804 8162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "0039d969e18240cca2fb882dcb844d85"
format_stamp: "Formatted at 2026-09-28 04:54:29 on dist-test-slave-m39l"
server_key: "c29249a98ec2ebe25d14b62679d2496b"
server_key_iv: "2772cc53e27c904eddaad54d090771e0"
server_key_version: "encryptionkey@0"
I20260928 04:54:29.593197 8162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0039d969e18240cca2fb882dcb844d85"
format_stamp: "Formatted at 2026-09-28 04:54:29 on dist-test-slave-m39l"
server_key: "c29249a98ec2ebe25d14b62679d2496b"
server_key_iv: "2772cc53e27c904eddaad54d090771e0"
server_key_version: "encryptionkey@0"
I20260928 04:54:29.594878 8162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 04:54:29.595777 8177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:29.596066 8162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:29.596302 8162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "0039d969e18240cca2fb882dcb844d85"
format_stamp: "Formatted at 2026-09-28 04:54:29 on dist-test-slave-m39l"
server_key: "c29249a98ec2ebe25d14b62679d2496b"
server_key_iv: "2772cc53e27c904eddaad54d090771e0"
server_key_version: "encryptionkey@0"
I20260928 04:54:29.596441 8162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:29.607518 8162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:29.609897 8162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:29.610093 8162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:29.610821 8162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:54:29.610888 8162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:29.610925 8162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:54:29.610953 8162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:29.618507 8162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34485
I20260928 04:54:29.618619 8292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34485 every 8 connection(s)
Sep 28 04:54:29 dist-test-slave-m39l krb5kdc[7905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:54:29.618993 8162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:54:29.622218 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.611367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39183 (local address 127.0.98.62:45175)
0928 04:54:29.611437 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:29.611440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:29.611643 (+ 203us) server_negotiation.cc:407] Connection header received
0928 04:54:29.612253 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:29.612254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:29.612267 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:29.612324 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:29.613394 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.614255 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.615978 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.616343 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.616422 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.619289 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:29.619291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:29.619292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:29.619302 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:29.620670 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:29.621188 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:29.621189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:29.621189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:29.621253 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:29.621533 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:29.621534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:29.621534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:29.621665 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:29.621745 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:29.622030 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 04:54:29.622121 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 04:54:29.622437 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.611340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45175 (local address 127.0.98.2:39183)
0928 04:54:29.611516 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:29.611554 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:29.612108 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:29.612464 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:29.612469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:29.612874 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:29.613198 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.613206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.614385 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.614386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.615877 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.615880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.616442 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.616442 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.616556 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.617148 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:29.617157 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:29.619161 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:29.620816 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:29.620824 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:29.620827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:29.621103 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:29.621334 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:29.621335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:29.621336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:29.621462 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:29.621736 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:29.621742 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:29.621896 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 04:54:29.622010 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 04:54:29.622185 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260928 04:54:29.622957 8293 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45175
I20260928 04:54:29.623087 8293 heartbeater.cc:464] Registering TS with master...
I20260928 04:54:29.623294 8293 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
I20260928 04:54:29.623731 7950 ts_manager.cc:194] Registered new tserver with Master: 0039d969e18240cca2fb882dcb844d85 (127.0.98.2:34485)
I20260928 04:54:29.624285 7950 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39183
I20260928 04:54:29.625969 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 8162
I20260928 04:54:29.626103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:54:29.626300 392 external_mini_cluster.cc:1428] Setting key e8b86383a4e8c1c8773e9c0c53f86341
I20260928 04:54:29.630892 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.624093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48215 (local address 127.0.98.62:45175)
0928 04:54:29.624141 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:29.624143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:29.624203 (+ 60us) server_negotiation.cc:407] Connection header received
0928 04:54:29.624277 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:29.624277 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:29.624288 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:29.624337 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:29.624795 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.625281 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.626085 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.626300 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.626395 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.627457 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:29.627458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:29.627459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:29.627467 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:29.629275 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:29.629728 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:29.629729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:29.629730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:29.629791 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:29.629980 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:29.629981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:29.629982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:29.630220 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:29.630295 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:29.630679 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 04:54:29.630767 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 04:54:29.630890 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.624016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45175 (local address 127.0.98.2:48215)
0928 04:54:29.624150 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:29.624161 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:29.624220 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:29.624338 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:29.624339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:29.624656 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:29.624717 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.624719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.625375 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.625376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.625970 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.625972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.626393 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.626394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.626466 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.626998 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:29.627005 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:29.627365 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:29.629454 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:29.629455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:29.629456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:29.629644 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:29.629864 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:29.629865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:29.629865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:29.629919 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:29.630286 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:29.630287 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:29.630330 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 04:54:29.630642 (+ 312us) client_negotiation.cc:240] Negotiation successful
0928 04:54:29.630768 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 04:54:29.658466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45175
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:54:29.744738 8302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:29.745023 8302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:29.745066 8302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:29.746321 8302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:29.746412 8302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:29.746449 8302 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:54:29.746487 8302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:54:29.748163 8302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:29.748481 8302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:29.748740 8302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:29.754444 8310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:29.754443 8307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:29.754547 8308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:29.754648 8302 server_base.cc:1034] running on GCE node
I20260928 04:54:29.754945 8302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:29.755214 8302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:29.756392 8302 hybrid_clock.cc:648] HybridClock initialized: now 1790571269756378 us; error 27 us; skew 500 ppm
Sep 28 04:54:29 dist-test-slave-m39l krb5kdc[7905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:54:29.759202 8302 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 04:54:29.759678 8302 webserver.cc:460] Webserver started at http://127.0.98.3:33149/ using document root <none> and password file <none>
I20260928 04:54:29.759930 8302 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:29.760002 8302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:29.760169 8302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:29.761302 8302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "734f02ecf4374b989840a2f278efedde"
format_stamp: "Formatted at 2026-09-28 04:54:29 on dist-test-slave-m39l"
server_key: "55da06088444fb02123ff5669984a26e"
server_key_iv: "d889092b51c4960874c0ebb97a96c21f"
server_key_version: "encryptionkey@0"
I20260928 04:54:29.761669 8302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "734f02ecf4374b989840a2f278efedde"
format_stamp: "Formatted at 2026-09-28 04:54:29 on dist-test-slave-m39l"
server_key: "55da06088444fb02123ff5669984a26e"
server_key_iv: "d889092b51c4960874c0ebb97a96c21f"
server_key_version: "encryptionkey@0"
I20260928 04:54:29.763206 8302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:54:29.764050 8317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:29.764219 8302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:54:29.764319 8302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "734f02ecf4374b989840a2f278efedde"
format_stamp: "Formatted at 2026-09-28 04:54:29 on dist-test-slave-m39l"
server_key: "55da06088444fb02123ff5669984a26e"
server_key_iv: "d889092b51c4960874c0ebb97a96c21f"
server_key_version: "encryptionkey@0"
I20260928 04:54:29.764408 8302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:29.778127 8302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:29.780562 8302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:29.780716 8302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:29.781196 8302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:54:29.781265 8302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:29.781322 8302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:54:29.781348 8302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 04:54:29 dist-test-slave-m39l krb5kdc[7905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:54:29.788810 8302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42111
I20260928 04:54:29.788898 8432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42111 every 8 connection(s)
I20260928 04:54:29.789255 8302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:54:29.791869 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.781810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42809 (local address 127.0.98.62:45175)
0928 04:54:29.781870 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:29.781873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:29.782168 (+ 295us) server_negotiation.cc:407] Connection header received
0928 04:54:29.782756 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:29.782757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:29.782798 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:29.782914 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:29.783916 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.784762 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.786380 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.786772 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.786858 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.789205 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:29.789207 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:29.789207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:29.789216 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:29.790659 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:29.791054 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:29.791055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:29.791055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:29.791106 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:29.791305 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:29.791305 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:29.791306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:29.791413 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:29.791465 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:29.791744 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 04:54:29.791807 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 04:54:29.792064 8326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.781835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45175 (local address 127.0.98.3:42809)
0928 04:54:29.782025 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:29.782069 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:29.782650 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:29.783016 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:29.783020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:29.783363 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:29.783808 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.783816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.784871 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.784872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.786294 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.786297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.786859 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.786860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.786967 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.787586 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:29.787596 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:29.789111 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:29.790731 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:29.790737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:29.790738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:29.790968 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:29.791150 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:29.791151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:29.791151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:29.791243 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:29.791482 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:29.791487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:29.791633 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 04:54:29.791718 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 04:54:29.791841 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":118,"threads_started":1}
I20260928 04:54:29.792572 8433 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45175
I20260928 04:54:29.792656 8433 heartbeater.cc:464] Registering TS with master...
I20260928 04:54:29.792850 8433 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
I20260928 04:54:29.793247 7949 ts_manager.cc:194] Registered new tserver with Master: 734f02ecf4374b989840a2f278efedde (127.0.98.3:42111)
I20260928 04:54:29.793823 7949 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42809
I20260928 04:54:29.794111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 8302
I20260928 04:54:29.794206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:54:29.794404 392 external_mini_cluster.cc:1428] Setting key 7ff02c22ae6ed1283815df4cb3ae8844
I20260928 04:54:29.795209 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:54:29.799808 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.793518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45175 (local address 127.0.98.3:49605)
0928 04:54:29.793668 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:29.793679 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:29.793737 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:29.793853 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:29.793854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:29.794062 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:29.794139 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.794141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.795056 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.795057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.795657 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.795660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.796132 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.796133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.796188 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.796648 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:29.796652 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:29.796952 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:29.798670 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:29.798671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:29.798671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:29.798841 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:29.799081 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:29.799082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:29.799082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:29.799137 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:29.799442 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:29.799443 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:29.799484 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 04:54:29.799604 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 04:54:29.799733 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260928 04:54:29.799865 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.793603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49605 (local address 127.0.98.62:45175)
0928 04:54:29.793646 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:29.793648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:29.793723 (+ 75us) server_negotiation.cc:407] Connection header received
0928 04:54:29.793789 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:29.793790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:29.793798 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:29.793837 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:29.794216 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.794960 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.795740 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.796051 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.796125 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.797041 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:29.797043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:29.797043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:29.797050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:29.798589 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:29.798948 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:29.798949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:29.798950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:29.799009 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:29.799199 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:29.799200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:29.799200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:29.799343 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:29.799423 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:29.799648 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 04:54:29.799732 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 04:54:29 dist-test-slave-m39l krb5kdc[7905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571269, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 04:54:29 dist-test-slave-m39l krb5kdc[7905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:29 dist-test-slave-m39l krb5kdc[7905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 04:54:29.818677 7999 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:29.812709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44654 (local address 127.0.98.62:45175)
0928 04:54:29.812780 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:29.812781 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:29.812790 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:54:29.812815 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:29.812816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:29.812824 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:29.812885 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:29.813477 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.813962 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.814966 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.815168 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.815278 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.818583 (+ 3305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44654: BlockingRecv error: recv got EOF from 127.0.0.1:44654 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 04:54:29 dist-test-slave-m39l krb5kdc[7905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571269, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:54:29.827167 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.819606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44670 (local address 127.0.98.62:45175)
0928 04:54:29.819680 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:29.819682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:29.819729 (+ 47us) server_negotiation.cc:407] Connection header received
0928 04:54:29.819796 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:29.819796 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:29.819804 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:29.819855 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:29.820320 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.820846 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.821613 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.821798 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.821857 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.824588 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:29.824590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:29.824590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:29.824597 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:29.826112 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:29.826421 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:29.826422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:29.826422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:29.826474 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:29.826689 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:29.826689 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:29.826690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:29.826808 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:29.826903 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:29.827039 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 04:54:29.827098 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 04:54:29.829134 7949 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:54:29.830976 7949 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:54:29.837963 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.833589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.0.98.2:34485)
0928 04:54:29.833789 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:29.833792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:29.834079 (+ 287us) server_negotiation.cc:407] Connection header received
0928 04:54:29.834152 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:29.834155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:29.834174 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:29.834249 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:54:29.834541 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.835610 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.837017 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.837525 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.837617 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.837698 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:29.837818 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 04:54:29.837856 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":100,"threads_started":1}
I20260928 04:54:29.837953 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.833402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34485 (local address 127.0.0.1:58010)
0928 04:54:29.834014 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:29.834025 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:29.834082 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:29.834348 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:29.834349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:29.834352 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:54:29.834428 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.834430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.835694 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.835694 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.836905 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.836907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.837626 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.837627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.837712 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.837727 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 04:54:29.837800 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 04:54:29.837863 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":5,"thread_start_us":84,"threads_started":1}
I20260928 04:54:29.838056 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.833381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42111 (local address 127.0.0.1:41580)
0928 04:54:29.833498 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:29.833543 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:29.833615 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:29.833982 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:29.833985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:29.833988 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:54:29.834089 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.834091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.835597 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.835598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.836905 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.836907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.837844 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.837845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.837904 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.837907 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:54:29.837952 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 04:54:29.837986 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":115,"thread_start_us":70,"threads_started":1}
I20260928 04:54:29.838171 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.833832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.0.98.1:40913)
0928 04:54:29.834016 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:29.834020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:29.834290 (+ 270us) server_negotiation.cc:407] Connection header received
0928 04:54:29.834340 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:29.834342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:29.834367 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:29.834485 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:54:29.834778 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.835495 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.836809 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.837569 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.837641 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.837714 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:29.838017 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 04:54:29.838060 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":94,"threads_started":1}
I20260928 04:54:29.838302 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.833608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40913 (local address 127.0.0.1:56722)
0928 04:54:29.834168 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:29.834177 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:29.834220 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:29.834613 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:29.834614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:29.834616 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:54:29.834687 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.834688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.835593 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.835594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.836711 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.836713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.837749 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.837750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.837848 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.837852 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:54:29.837911 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 04:54:29.837964 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":48,"thread_start_us":71,"threads_started":1}
I20260928 04:54:29.838385 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.833570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.0.98.3:42111)
0928 04:54:29.833766 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:29.833771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:29.833783 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:54:29.833827 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:29.833829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:29.833845 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:29.833951 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:54:29.834185 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.835498 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.836992 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.837686 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.837748 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.837813 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:29.838160 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 04:54:29.838208 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":89,"threads_started":1}
I20260928 04:54:29.839073 8087 tablet_service.cc:1467] Processing CreateTablet for tablet b4958b36894448e9929c492ae277e7c6 (DEFAULT_TABLE table=test-table [id=f559664f8b9b448e81118517c511bdea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:54:29.839097 8367 tablet_service.cc:1467] Processing CreateTablet for tablet b4958b36894448e9929c492ae277e7c6 (DEFAULT_TABLE table=test-table [id=f559664f8b9b448e81118517c511bdea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:54:29.839288 8227 tablet_service.cc:1467] Processing CreateTablet for tablet b4958b36894448e9929c492ae277e7c6 (DEFAULT_TABLE table=test-table [id=f559664f8b9b448e81118517c511bdea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:54:29.839418 8087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4958b36894448e9929c492ae277e7c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:29.839435 8367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4958b36894448e9929c492ae277e7c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:29.839577 8227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4958b36894448e9929c492ae277e7c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:29.842671 8461 tablet_bootstrap.cc:492] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650: Bootstrap starting.
I20260928 04:54:29.843295 8462 tablet_bootstrap.cc:492] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde: Bootstrap starting.
I20260928 04:54:29.843470 8461 tablet_bootstrap.cc:654] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:29.843677 8463 tablet_bootstrap.cc:492] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85: Bootstrap starting.
I20260928 04:54:29.843813 8461 log.cc:824] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650: Log is configured to *not* fsync() on all Append() calls
I20260928 04:54:29.843962 8462 tablet_bootstrap.cc:654] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:29.844259 8462 log.cc:824] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde: Log is configured to *not* fsync() on all Append() calls
I20260928 04:54:29.844331 8463 tablet_bootstrap.cc:654] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:29.844663 8461 tablet_bootstrap.cc:492] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650: No bootstrap required, opened a new log
I20260928 04:54:29.844643 8463 log.cc:824] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85: Log is configured to *not* fsync() on all Append() calls
I20260928 04:54:29.844758 8461 ts_tablet_manager.cc:1397] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:54:29.845052 8462 tablet_bootstrap.cc:492] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde: No bootstrap required, opened a new log
I20260928 04:54:29.845126 8462 ts_tablet_manager.cc:1397] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:54:29.845537 8463 tablet_bootstrap.cc:492] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85: No bootstrap required, opened a new log
I20260928 04:54:29.845666 8463 ts_tablet_manager.cc:1397] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:54:29.846674 8462 raft_consensus.cc:348] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } }
I20260928 04:54:29.846814 8462 raft_consensus.cc:374] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:29.846850 8462 raft_consensus.cc:729] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734f02ecf4374b989840a2f278efedde, State: Initialized, Role: FOLLOWER
I20260928 04:54:29.846974 8462 consensus_queue.cc:260] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } }
I20260928 04:54:29.847263 8462 ts_tablet_manager.cc:1428] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:54:29.847281 8433 heartbeater.cc:503] Master 127.0.98.62:45175 was elected leader, sending a full tablet report...
I20260928 04:54:29.847123 8461 raft_consensus.cc:348] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } }
I20260928 04:54:29.847656 8461 raft_consensus.cc:374] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:29.847708 8461 raft_consensus.cc:729] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f9ec8981c54a28b7ea086466a70650, State: Initialized, Role: FOLLOWER
I20260928 04:54:29.847831 8461 consensus_queue.cc:260] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } }
I20260928 04:54:29.848183 8461 ts_tablet_manager.cc:1428] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:54:29.848218 8153 heartbeater.cc:503] Master 127.0.98.62:45175 was elected leader, sending a full tablet report...
I20260928 04:54:29.848158 8463 raft_consensus.cc:348] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } }
I20260928 04:54:29.848311 8463 raft_consensus.cc:374] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:29.848347 8463 raft_consensus.cc:729] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0039d969e18240cca2fb882dcb844d85, State: Initialized, Role: FOLLOWER
I20260928 04:54:29.848466 8463 consensus_queue.cc:260] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } }
I20260928 04:54:29.848758 8463 ts_tablet_manager.cc:1428] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:54:29.848894 8293 heartbeater.cc:503] Master 127.0.98.62:45175 was elected leader, sending a full tablet report...
W20260928 04:54:29.869668 8294 tablet.cc:2369] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:54:29.954217 8154 tablet.cc:2369] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:54:29.983948 8468 raft_consensus.cc:482] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:54:29.984131 8468 raft_consensus.cc:504] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } }
I20260928 04:54:29.984642 8467 raft_consensus.cc:482] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:54:29.984751 8467 raft_consensus.cc:504] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } }
I20260928 04:54:29.985093 8467 leader_election.cc:288] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0039d969e18240cca2fb882dcb844d85 (127.0.98.2:34485), 59f9ec8981c54a28b7ea086466a70650 (127.0.98.1:40913)
I20260928 04:54:29.984567 8468 leader_election.cc:288] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 734f02ecf4374b989840a2f278efedde (127.0.98.3:42111), 0039d969e18240cca2fb882dcb844d85 (127.0.98.2:34485)
I20260928 04:54:29.988555 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.984964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45195 (local address 127.0.98.3:42111)
0928 04:54:29.985005 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:29.985007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:29.985099 (+ 92us) server_negotiation.cc:407] Connection header received
0928 04:54:29.985205 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:29.985206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:29.985217 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:29.985247 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:54:29.985522 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.986232 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.987688 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.988187 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.988228 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.988239 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:29.988444 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 04:54:29.988486 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:54:29.988592 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.984921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51181 (local address 127.0.98.2:34485)
0928 04:54:29.984969 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:29.984972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:29.985036 (+ 64us) server_negotiation.cc:407] Connection header received
0928 04:54:29.985142 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:29.985143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:29.985156 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:29.985189 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:54:29.985423 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.986160 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.987691 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.988227 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.988267 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.988276 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:29.988471 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 04:54:29.988510 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:54:29.988835 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.985416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34299 (local address 127.0.98.1:40913)
0928 04:54:29.985723 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:29.985725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:29.985733 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:54:29.985752 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:29.985752 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:29.985762 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:29.985815 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:54:29.985992 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.986658 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.987962 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.988476 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.988511 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.988530 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:29.988752 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 04:54:29.988776 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290}
I20260928 04:54:29.988837 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.984764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34485 (local address 127.0.98.1:51181)
0928 04:54:29.984937 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:29.984950 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:29.985048 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:29.985267 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:29.985268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:29.985271 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:54:29.985341 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.985344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.986247 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.986247 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.987591 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.987594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.988328 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.988329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.988393 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.988396 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:54:29.988708 (+ 312us) client_negotiation.cc:240] Negotiation successful
0928 04:54:29.988761 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":37,"thread_start_us":82,"threads_started":1}
I20260928 04:54:29.988943 8326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.985170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34485 (local address 127.0.98.3:51045)
0928 04:54:29.985212 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:29.985221 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:29.985254 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:29.985764 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:29.985765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:29.985766 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:54:29.985819 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.985820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.986732 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.986733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.987869 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.987870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.988700 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.988701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.988785 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.988788 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:54:29.988815 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 04:54:29.988855 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":1}
I20260928 04:54:29.989128 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.985259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40913 (local address 127.0.98.3:34299)
0928 04:54:29.985476 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:29.985487 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:29.985526 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:29.985854 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:29.985855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:29.985857 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:54:29.985925 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.985926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.986745 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.986746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.987879 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.987881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.988586 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.988586 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.988669 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.988672 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:54:29.988947 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 04:54:29.988994 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260928 04:54:29.989219 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.985351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51045 (local address 127.0.98.2:34485)
0928 04:54:29.985638 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:29.985640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:29.985647 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:54:29.985668 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:29.985669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:29.985677 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:29.985874 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:54:29.985902 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.986634 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.987956 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.988605 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.989064 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.989078 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:29.989098 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 04:54:29.989136 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":77,"threads_started":1}
I20260928 04:54:29.989256 8247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4958b36894448e9929c492ae277e7c6" candidate_uuid: "59f9ec8981c54a28b7ea086466a70650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0039d969e18240cca2fb882dcb844d85" is_pre_election: true
I20260928 04:54:29.989385 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:29.984765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42111 (local address 127.0.98.1:45195)
0928 04:54:29.985030 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:29.985041 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:29.985126 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:29.985317 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:29.985318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:29.985320 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:54:29.985397 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.985399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.986321 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.986321 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.987589 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:29.987592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:29.988295 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:29.988295 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:29.988366 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:29.988370 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:54:29.989254 (+ 884us) client_negotiation.cc:240] Negotiation successful
0928 04:54:29.989306 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":9,"thread_start_us":84,"threads_started":1}
I20260928 04:54:29.989409 8247 raft_consensus.cc:2454] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f9ec8981c54a28b7ea086466a70650 in term 0.
I20260928 04:54:29.989446 8107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4958b36894448e9929c492ae277e7c6" candidate_uuid: "734f02ecf4374b989840a2f278efedde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f9ec8981c54a28b7ea086466a70650" is_pre_election: true
I20260928 04:54:29.989573 8107 raft_consensus.cc:2454] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 734f02ecf4374b989840a2f278efedde in term 0.
I20260928 04:54:29.989639 8041 leader_election.cc:302] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0039d969e18240cca2fb882dcb844d85, 59f9ec8981c54a28b7ea086466a70650; no voters:
I20260928 04:54:29.989668 8246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4958b36894448e9929c492ae277e7c6" candidate_uuid: "734f02ecf4374b989840a2f278efedde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0039d969e18240cca2fb882dcb844d85" is_pre_election: true
I20260928 04:54:29.989730 8246 raft_consensus.cc:2454] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 734f02ecf4374b989840a2f278efedde in term 0.
I20260928 04:54:29.989791 8320 leader_election.cc:302] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f9ec8981c54a28b7ea086466a70650, 734f02ecf4374b989840a2f278efedde; no voters:
I20260928 04:54:29.989897 8467 raft_consensus.cc:2781] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:54:29.989956 8467 raft_consensus.cc:482] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:54:29.989991 8467 raft_consensus.cc:3037] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:29.990028 8387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4958b36894448e9929c492ae277e7c6" candidate_uuid: "59f9ec8981c54a28b7ea086466a70650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734f02ecf4374b989840a2f278efedde" is_pre_election: true
I20260928 04:54:29.990157 8468 raft_consensus.cc:2781] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:54:29.990233 8468 raft_consensus.cc:482] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:54:29.990266 8468 raft_consensus.cc:3037] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:29.990955 8467 raft_consensus.cc:504] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } }
I20260928 04:54:29.991148 8467 leader_election.cc:288] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [CANDIDATE]: Term 1 election: Requested vote from peers 0039d969e18240cca2fb882dcb844d85 (127.0.98.2:34485), 59f9ec8981c54a28b7ea086466a70650 (127.0.98.1:40913)
I20260928 04:54:29.991118 8468 raft_consensus.cc:504] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } }
I20260928 04:54:29.991156 8387 raft_consensus.cc:2379] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59f9ec8981c54a28b7ea086466a70650 in current term 1: Already voted for candidate 734f02ecf4374b989840a2f278efedde in this term.
I20260928 04:54:29.991297 8246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4958b36894448e9929c492ae277e7c6" candidate_uuid: "734f02ecf4374b989840a2f278efedde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0039d969e18240cca2fb882dcb844d85"
I20260928 04:54:29.991364 8247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4958b36894448e9929c492ae277e7c6" candidate_uuid: "59f9ec8981c54a28b7ea086466a70650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0039d969e18240cca2fb882dcb844d85"
I20260928 04:54:29.991371 8246 raft_consensus.cc:3037] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:29.991415 8386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4958b36894448e9929c492ae277e7c6" candidate_uuid: "59f9ec8981c54a28b7ea086466a70650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734f02ecf4374b989840a2f278efedde"
I20260928 04:54:29.991653 8107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4958b36894448e9929c492ae277e7c6" candidate_uuid: "734f02ecf4374b989840a2f278efedde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f9ec8981c54a28b7ea086466a70650"
I20260928 04:54:29.991693 8468 leader_election.cc:288] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [CANDIDATE]: Term 1 election: Requested vote from peers 734f02ecf4374b989840a2f278efedde (127.0.98.3:42111), 0039d969e18240cca2fb882dcb844d85 (127.0.98.2:34485)
I20260928 04:54:29.991734 8107 raft_consensus.cc:2379] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 734f02ecf4374b989840a2f278efedde in current term 1: Already voted for candidate 59f9ec8981c54a28b7ea086466a70650 in this term.
I20260928 04:54:29.991933 8039 leader_election.cc:302] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f9ec8981c54a28b7ea086466a70650; no voters: 0039d969e18240cca2fb882dcb844d85, 734f02ecf4374b989840a2f278efedde
I20260928 04:54:29.992103 8468 raft_consensus.cc:2726] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 04:54:29.992370 8246 raft_consensus.cc:2454] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 734f02ecf4374b989840a2f278efedde in term 1.
I20260928 04:54:29.992560 8321 leader_election.cc:302] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0039d969e18240cca2fb882dcb844d85, 734f02ecf4374b989840a2f278efedde; no voters: 59f9ec8981c54a28b7ea086466a70650
I20260928 04:54:29.992666 8467 raft_consensus.cc:2781] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:54:29.992816 8467 raft_consensus.cc:686] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [term 1 LEADER]: Becoming Leader. State: Replica: 734f02ecf4374b989840a2f278efedde, State: Running, Role: LEADER
I20260928 04:54:29.992889 8467 consensus_queue.cc:237] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } }
I20260928 04:54:29.993744 7949 catalog_manager.cc:5495] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde reported cstate change: term changed from 0 to 1, leader changed from <none> to 734f02ecf4374b989840a2f278efedde (127.0.98.3). New cstate: current_term: 1 leader_uuid: "734f02ecf4374b989840a2f278efedde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } health_report { overall_health: UNKNOWN } } }
I20260928 04:54:30.021646 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:30.018832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41592 (local address 127.0.98.3:42111)
0928 04:54:30.018913 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:30.018915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:30.018925 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:54:30.018963 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:30.018964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:30.018982 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:30.019056 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:54:30.019212 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.019858 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.020672 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.020915 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.020980 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:30.021111 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:54:30.021239 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:54:30.021306 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:30.021480 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 04:54:30.021503 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 04:54:30.023777 8246 raft_consensus.cc:1261] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85 [term 1 FOLLOWER]: Refusing update from remote peer 734f02ecf4374b989840a2f278efedde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:54:30.023775 8107 raft_consensus.cc:1261] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [term 1 FOLLOWER]: Refusing update from remote peer 734f02ecf4374b989840a2f278efedde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:54:30.024113 8474 consensus_queue.cc:1035] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:54:30.024260 8467 consensus_queue.cc:1035] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [LEADER]: Connected to new peer: Peer: permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:54:30.026799 8477 mvcc.cc:205] Tried to move back new op lower bound from 7334179922014711808 to 7334179921891557376. Current Snapshot: MvccSnapshot[applied={T|T < 7334179922014711808}]
I20260928 04:54:30.027933 8478 mvcc.cc:205] Tried to move back new op lower bound from 7334179922014711808 to 7334179921891557376. Current Snapshot: MvccSnapshot[applied={T|T < 7334179922014711808}]
I20260928 04:54:30.028198 8479 mvcc.cc:205] Tried to move back new op lower bound from 7334179922014711808 to 7334179921891557376. Current Snapshot: MvccSnapshot[applied={T|T < 7334179922014711808}]
I20260928 04:54:30.032660 7950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:54:30.032796 7950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:54:30.033535 7950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78: Sending DeleteTablet for 3 replicas of tablet b4958b36894448e9929c492ae277e7c6
I20260928 04:54:30.033875 8367 tablet_service.cc:1514] Processing DeleteTablet for tablet b4958b36894448e9929c492ae277e7c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:54:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41580
I20260928 04:54:30.033881 8227 tablet_service.cc:1514] Processing DeleteTablet for tablet b4958b36894448e9929c492ae277e7c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:54:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58010
I20260928 04:54:30.034114 8488 tablet_replica.cc:330] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85: stopping tablet replica
I20260928 04:54:30.034235 8488 raft_consensus.cc:2229] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:54:30.034305 8087 tablet_service.cc:1514] Processing DeleteTablet for tablet b4958b36894448e9929c492ae277e7c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:54:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56722
I20260928 04:54:30.034368 8488 raft_consensus.cc:2258] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:54:30.034544 8487 tablet_replica.cc:330] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde: stopping tablet replica
I20260928 04:54:30.034641 8487 raft_consensus.cc:2229] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:54:30.034765 8487 raft_consensus.cc:2258] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:54:30.034735 7950 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:54:30.034897 7950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:54:30.035164 8488 ts_tablet_manager.cc:1905] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:54:30.035495 8489 tablet_replica.cc:330] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650: stopping tablet replica
I20260928 04:54:30.035612 8489 raft_consensus.cc:2229] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:54:30.035732 8489 raft_consensus.cc:2258] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:54:30.036171 8489 ts_tablet_manager.cc:1905] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:54:30.036649 8227 tablet_service.cc:1467] Processing CreateTablet for tablet 43084050970041528e1a23f6690f3b7b (DEFAULT_TABLE table=test-table [id=e70e58e7d1f24bfda0a5514e317549ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:54:30.036768 8227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43084050970041528e1a23f6690f3b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:30.037329 8488 ts_tablet_manager.cc:1918] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:54:30.037485 8488 log.cc:1196] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/b4958b36894448e9929c492ae277e7c6
I20260928 04:54:30.037828 8488 ts_tablet_manager.cc:1939] T b4958b36894448e9929c492ae277e7c6 P 0039d969e18240cca2fb882dcb844d85: Deleting consensus metadata
I20260928 04:54:30.037886 8489 ts_tablet_manager.cc:1918] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:54:30.037962 8489 log.cc:1196] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/wal/wals/b4958b36894448e9929c492ae277e7c6
I20260928 04:54:30.038241 8489 ts_tablet_manager.cc:1939] T b4958b36894448e9929c492ae277e7c6 P 59f9ec8981c54a28b7ea086466a70650: Deleting consensus metadata
I20260928 04:54:30.038233 8463 tablet_bootstrap.cc:492] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85: Bootstrap starting.
I20260928 04:54:30.038697 8463 tablet_bootstrap.cc:654] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:30.038841 7936 catalog_manager.cc:4841] TS 59f9ec8981c54a28b7ea086466a70650 (127.0.98.1:40913): tablet b4958b36894448e9929c492ae277e7c6 (table test-table [id=f559664f8b9b448e81118517c511bdea]) successfully deleted
I20260928 04:54:30.038942 8087 tablet_service.cc:1467] Processing CreateTablet for tablet 43084050970041528e1a23f6690f3b7b (DEFAULT_TABLE table=test-table [id=e70e58e7d1f24bfda0a5514e317549ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:54:30.039041 8087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43084050970041528e1a23f6690f3b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:30.039075 8367 tablet_service.cc:1467] Processing CreateTablet for tablet 43084050970041528e1a23f6690f3b7b (DEFAULT_TABLE table=test-table [id=e70e58e7d1f24bfda0a5514e317549ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:54:30.039172 8367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43084050970041528e1a23f6690f3b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:30.039224 8487 ts_tablet_manager.cc:1905] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:54:30.040227 8461 tablet_bootstrap.cc:492] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650: Bootstrap starting.
I20260928 04:54:30.040568 8462 tablet_bootstrap.cc:492] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde: Bootstrap starting.
I20260928 04:54:30.040627 8461 tablet_bootstrap.cc:654] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:30.041002 8487 ts_tablet_manager.cc:1918] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:54:30.041040 8462 tablet_bootstrap.cc:654] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:30.041067 8487 log.cc:1196] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/b4958b36894448e9929c492ae277e7c6
I20260928 04:54:30.041316 8487 ts_tablet_manager.cc:1939] T b4958b36894448e9929c492ae277e7c6 P 734f02ecf4374b989840a2f278efedde: Deleting consensus metadata
I20260928 04:54:30.041419 7937 catalog_manager.cc:4841] TS 0039d969e18240cca2fb882dcb844d85 (127.0.98.2:34485): tablet b4958b36894448e9929c492ae277e7c6 (table test-table [id=f559664f8b9b448e81118517c511bdea]) successfully deleted
I20260928 04:54:30.041792 8462 tablet_bootstrap.cc:492] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde: No bootstrap required, opened a new log
I20260928 04:54:30.041839 8462 ts_tablet_manager.cc:1397] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:30.041867 7935 catalog_manager.cc:4841] TS 734f02ecf4374b989840a2f278efedde (127.0.98.3:42111): tablet b4958b36894448e9929c492ae277e7c6 (table test-table [id=f559664f8b9b448e81118517c511bdea]) successfully deleted
I20260928 04:54:30.042002 8462 raft_consensus.cc:348] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.042096 8462 raft_consensus.cc:374] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:30.042124 8462 raft_consensus.cc:729] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734f02ecf4374b989840a2f278efedde, State: Initialized, Role: FOLLOWER
I20260928 04:54:30.042178 8462 consensus_queue.cc:260] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.042268 8462 ts_tablet_manager.cc:1428] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:30.042392 8463 tablet_bootstrap.cc:492] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85: No bootstrap required, opened a new log
I20260928 04:54:30.042438 8463 ts_tablet_manager.cc:1397] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 04:54:30.042469 8461 tablet_bootstrap.cc:492] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650: No bootstrap required, opened a new log
I20260928 04:54:30.042536 8461 ts_tablet_manager.cc:1397] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:54:30.042611 8463 raft_consensus.cc:348] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.042682 8463 raft_consensus.cc:374] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:30.042703 8463 raft_consensus.cc:729] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0039d969e18240cca2fb882dcb844d85, State: Initialized, Role: FOLLOWER
I20260928 04:54:30.042692 8461 raft_consensus.cc:348] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.042747 8463 consensus_queue.cc:260] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.042759 8461 raft_consensus.cc:374] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:30.042801 8461 raft_consensus.cc:729] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f9ec8981c54a28b7ea086466a70650, State: Initialized, Role: FOLLOWER
I20260928 04:54:30.042833 8463 ts_tablet_manager.cc:1428] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 04:54:30.042845 8461 consensus_queue.cc:260] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.042929 8461 ts_tablet_manager.cc:1428] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:30.200616 8469 raft_consensus.cc:482] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:54:30.200719 8469 raft_consensus.cc:504] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.201056 8469 leader_election.cc:288] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f9ec8981c54a28b7ea086466a70650 (127.0.98.1:40913), 734f02ecf4374b989840a2f278efedde (127.0.98.3:42111)
I20260928 04:54:30.204535 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:30.201345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56061 (local address 127.0.98.3:42111)
0928 04:54:30.201401 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:30.201403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:30.201594 (+ 191us) server_negotiation.cc:407] Connection header received
0928 04:54:30.201662 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:30.201663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:30.201681 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:30.201749 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:54:30.201969 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.202626 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.203739 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.204131 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.204190 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:30.204199 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:30.204405 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 04:54:30.204450 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:54:30.204579 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:30.201497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59677 (local address 127.0.98.1:40913)
0928 04:54:30.201567 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:30.201570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:30.201765 (+ 195us) server_negotiation.cc:407] Connection header received
0928 04:54:30.201797 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:30.201798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:30.201808 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:30.201898 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:54:30.202078 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.202705 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.203740 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.204150 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.204250 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:30.204262 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:30.204472 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 04:54:30.204517 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 04:54:30.204607 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:30.201271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40913 (local address 127.0.98.2:59677)
0928 04:54:30.201688 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:30.201697 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:30.201749 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:30.201907 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:30.201907 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:30.201909 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:54:30.201989 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:30.201991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.202810 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.202811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:30.203634 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:30.203636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.204266 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.204267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:30.204348 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:30.204352 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:54:30.204436 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 04:54:30.204485 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260928 04:54:30.204689 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:30.201271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42111 (local address 127.0.98.2:56061)
0928 04:54:30.201477 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:30.201488 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:30.201593 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:30.201793 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:30.201795 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:30.201798 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:54:30.201885 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:30.201888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.202710 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.202711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:30.203631 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:30.203633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.204222 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.204223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:30.204317 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:30.204321 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:54:30.204377 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 04:54:30.204439 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":134,"threads_started":1}
I20260928 04:54:30.204926 8107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43084050970041528e1a23f6690f3b7b" candidate_uuid: "0039d969e18240cca2fb882dcb844d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f9ec8981c54a28b7ea086466a70650" is_pre_election: true
I20260928 04:54:30.205031 8107 raft_consensus.cc:2454] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0039d969e18240cca2fb882dcb844d85 in term 0.
I20260928 04:54:30.204979 8386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43084050970041528e1a23f6690f3b7b" candidate_uuid: "0039d969e18240cca2fb882dcb844d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734f02ecf4374b989840a2f278efedde" is_pre_election: true
I20260928 04:54:30.205090 8386 raft_consensus.cc:2454] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0039d969e18240cca2fb882dcb844d85 in term 0.
I20260928 04:54:30.205228 8180 leader_election.cc:302] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0039d969e18240cca2fb882dcb844d85, 59f9ec8981c54a28b7ea086466a70650; no voters:
I20260928 04:54:30.205389 8469 raft_consensus.cc:2781] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:54:30.205437 8469 raft_consensus.cc:482] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:54:30.205472 8469 raft_consensus.cc:3037] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:30.206198 8469 raft_consensus.cc:504] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.206377 8469 leader_election.cc:288] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [CANDIDATE]: Term 1 election: Requested vote from peers 59f9ec8981c54a28b7ea086466a70650 (127.0.98.1:40913), 734f02ecf4374b989840a2f278efedde (127.0.98.3:42111)
I20260928 04:54:30.206607 8107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43084050970041528e1a23f6690f3b7b" candidate_uuid: "0039d969e18240cca2fb882dcb844d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f9ec8981c54a28b7ea086466a70650"
I20260928 04:54:30.206638 8386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43084050970041528e1a23f6690f3b7b" candidate_uuid: "0039d969e18240cca2fb882dcb844d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734f02ecf4374b989840a2f278efedde"
I20260928 04:54:30.206699 8107 raft_consensus.cc:3037] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:30.206713 8386 raft_consensus.cc:3037] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:30.207567 8107 raft_consensus.cc:2454] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0039d969e18240cca2fb882dcb844d85 in term 1.
I20260928 04:54:30.207567 8386 raft_consensus.cc:2454] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0039d969e18240cca2fb882dcb844d85 in term 1.
I20260928 04:54:30.207791 8180 leader_election.cc:302] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0039d969e18240cca2fb882dcb844d85, 59f9ec8981c54a28b7ea086466a70650; no voters:
I20260928 04:54:30.207938 8469 raft_consensus.cc:2781] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:54:30.208098 8469 raft_consensus.cc:686] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [term 1 LEADER]: Becoming Leader. State: Replica: 0039d969e18240cca2fb882dcb844d85, State: Running, Role: LEADER
I20260928 04:54:30.208189 8469 consensus_queue.cc:237] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.208941 7949 catalog_manager.cc:5495] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0039d969e18240cca2fb882dcb844d85 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0039d969e18240cca2fb882dcb844d85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } health_report { overall_health: UNKNOWN } } }
Sep 28 04:54:30 dist-test-slave-m39l krb5kdc[7905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:54:30.258915 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:30.251395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45175 (local address 127.0.0.1:44684)
0928 04:54:30.251564 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:30.251574 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:30.251663 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:30.251833 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:30.251833 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:30.252194 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:30.252247 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:30.252250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.252974 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.252976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:30.253507 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:30.253508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.253887 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.253888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:30.253946 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:30.254555 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:30.254566 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:30.255883 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:30.257522 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:30.257530 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:30.257532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:30.257810 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:30.258010 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:30.258011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:30.258011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:30.258069 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:30.258325 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:30.258332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:30.258380 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 04:54:30.258471 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 04:54:30.258760 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":90,"threads_started":1}
I20260928 04:54:30.258997 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:30.251423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.0.98.62:45175)
0928 04:54:30.251496 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:30.251498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:30.251644 (+ 146us) server_negotiation.cc:407] Connection header received
0928 04:54:30.251727 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:30.251728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:30.251738 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:30.251815 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:30.252334 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.252874 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.253604 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.253784 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.253874 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:30.255974 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:30.255975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:30.255976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:30.255983 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:30.257412 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:30.257881 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:30.257883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:30.257883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:30.257942 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:30.258139 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:30.258140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:30.258140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:30.258259 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:30.258335 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:30.258554 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 04:54:30.258870 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":225,"server-negotiator.queue_time_us":35}
I20260928 04:54:30.261394 7949 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44684:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 04:54:30.261648 7949 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:54:30.263432 8227 tablet_service.cc:1467] Processing CreateTablet for tablet bd8eeaf5c0a54adbb79735af42db007d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7540a2256bf436fa65784a92c70eb31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:54:30.263424 8087 tablet_service.cc:1467] Processing CreateTablet for tablet bd8eeaf5c0a54adbb79735af42db007d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7540a2256bf436fa65784a92c70eb31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:54:30.263440 8367 tablet_service.cc:1467] Processing CreateTablet for tablet bd8eeaf5c0a54adbb79735af42db007d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7540a2256bf436fa65784a92c70eb31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:54:30.263613 8227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8eeaf5c0a54adbb79735af42db007d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:30.263613 8087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8eeaf5c0a54adbb79735af42db007d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:30.263626 8367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8eeaf5c0a54adbb79735af42db007d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:30.265015 8463 tablet_bootstrap.cc:492] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85: Bootstrap starting.
I20260928 04:54:30.265151 8462 tablet_bootstrap.cc:492] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde: Bootstrap starting.
I20260928 04:54:30.265482 8463 tablet_bootstrap.cc:654] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:30.265481 8462 tablet_bootstrap.cc:654] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:30.265786 8461 tablet_bootstrap.cc:492] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650: Bootstrap starting.
I20260928 04:54:30.266208 8461 tablet_bootstrap.cc:654] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:30.266248 8462 tablet_bootstrap.cc:492] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde: No bootstrap required, opened a new log
I20260928 04:54:30.266295 8462 ts_tablet_manager.cc:1397] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:30.266458 8462 raft_consensus.cc:348] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.266614 8462 raft_consensus.cc:374] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:30.266640 8462 raft_consensus.cc:729] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734f02ecf4374b989840a2f278efedde, State: Initialized, Role: FOLLOWER
I20260928 04:54:30.266695 8462 consensus_queue.cc:260] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.266809 8462 ts_tablet_manager.cc:1428] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:54:30.266848 8463 tablet_bootstrap.cc:492] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85: No bootstrap required, opened a new log
I20260928 04:54:30.266822 8484 tablet_replica.cc:439] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } }
I20260928 04:54:30.266887 8484 tablet_replica.cc:442] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:54:30.266893 8463 ts_tablet_manager.cc:1397] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:54:30.267040 8463 raft_consensus.cc:348] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.267100 8463 raft_consensus.cc:374] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:30.267143 8463 raft_consensus.cc:729] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0039d969e18240cca2fb882dcb844d85, State: Initialized, Role: FOLLOWER
I20260928 04:54:30.267189 8463 consensus_queue.cc:260] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.267287 8463 ts_tablet_manager.cc:1428] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 04:54:30.267345 8469 tablet_replica.cc:439] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } }
I20260928 04:54:30.267457 8469 tablet_replica.cc:442] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:54:30.267858 8461 tablet_bootstrap.cc:492] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650: No bootstrap required, opened a new log
I20260928 04:54:30.267908 8461 ts_tablet_manager.cc:1397] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:54:30.268059 8461 raft_consensus.cc:348] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.268152 8461 raft_consensus.cc:374] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:30.268184 8461 raft_consensus.cc:729] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f9ec8981c54a28b7ea086466a70650, State: Initialized, Role: FOLLOWER
I20260928 04:54:30.268236 8461 consensus_queue.cc:260] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.268321 8461 ts_tablet_manager.cc:1428] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:30.268376 8468 tablet_replica.cc:439] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } }
I20260928 04:54:30.268479 8468 tablet_replica.cc:442] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 04:54:30.290463 8434 tablet.cc:2369] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:54:30.359390 8468 raft_consensus.cc:482] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:54:30.359485 8468 raft_consensus.cc:504] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.359676 8468 leader_election.cc:288] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0039d969e18240cca2fb882dcb844d85 (127.0.98.2:34485), 734f02ecf4374b989840a2f278efedde (127.0.98.3:42111)
I20260928 04:54:30.360041 8246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8eeaf5c0a54adbb79735af42db007d" candidate_uuid: "59f9ec8981c54a28b7ea086466a70650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0039d969e18240cca2fb882dcb844d85" is_pre_election: true
I20260928 04:54:30.360041 8386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8eeaf5c0a54adbb79735af42db007d" candidate_uuid: "59f9ec8981c54a28b7ea086466a70650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734f02ecf4374b989840a2f278efedde" is_pre_election: true
I20260928 04:54:30.360149 8386 raft_consensus.cc:2454] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f9ec8981c54a28b7ea086466a70650 in term 0.
I20260928 04:54:30.360149 8246 raft_consensus.cc:2454] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f9ec8981c54a28b7ea086466a70650 in term 0.
I20260928 04:54:30.360373 8041 leader_election.cc:302] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0039d969e18240cca2fb882dcb844d85, 59f9ec8981c54a28b7ea086466a70650; no voters:
I20260928 04:54:30.360492 8468 raft_consensus.cc:2781] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:54:30.360549 8468 raft_consensus.cc:482] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:54:30.360581 8468 raft_consensus.cc:3037] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:30.361313 8468 raft_consensus.cc:504] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.361486 8468 leader_election.cc:288] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [CANDIDATE]: Term 1 election: Requested vote from peers 0039d969e18240cca2fb882dcb844d85 (127.0.98.2:34485), 734f02ecf4374b989840a2f278efedde (127.0.98.3:42111)
I20260928 04:54:30.361678 8386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8eeaf5c0a54adbb79735af42db007d" candidate_uuid: "59f9ec8981c54a28b7ea086466a70650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734f02ecf4374b989840a2f278efedde"
I20260928 04:54:30.361697 8246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8eeaf5c0a54adbb79735af42db007d" candidate_uuid: "59f9ec8981c54a28b7ea086466a70650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0039d969e18240cca2fb882dcb844d85"
I20260928 04:54:30.361792 8386 raft_consensus.cc:3037] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:30.361824 8246 raft_consensus.cc:3037] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:30.362604 8386 raft_consensus.cc:2454] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f9ec8981c54a28b7ea086466a70650 in term 1.
I20260928 04:54:30.362676 8246 raft_consensus.cc:2454] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f9ec8981c54a28b7ea086466a70650 in term 1.
I20260928 04:54:30.362797 8039 leader_election.cc:302] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f9ec8981c54a28b7ea086466a70650, 734f02ecf4374b989840a2f278efedde; no voters:
I20260928 04:54:30.362900 8468 raft_consensus.cc:2781] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:54:30.363070 8468 raft_consensus.cc:686] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [term 1 LEADER]: Becoming Leader. State: Replica: 59f9ec8981c54a28b7ea086466a70650, State: Running, Role: LEADER
I20260928 04:54:30.363189 8468 consensus_queue.cc:237] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.363682 8506 tablet_replica.cc:439] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650: TxnStatusTablet state changed. Reason: New leader 59f9ec8981c54a28b7ea086466a70650. Latest consensus state: current_term: 1 leader_uuid: "59f9ec8981c54a28b7ea086466a70650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } }
I20260928 04:54:30.363816 8506 tablet_replica.cc:442] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:54:30.364063 8508 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 04:54:30.364192 8508 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 04:54:30.364204 7950 catalog_manager.cc:5495] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f9ec8981c54a28b7ea086466a70650 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "59f9ec8981c54a28b7ea086466a70650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } health_report { overall_health: UNKNOWN } } }
I20260928 04:54:30.387445 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:30.383987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40913 (local address 127.0.0.1:56738)
0928 04:54:30.384073 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:30.384109 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:30.384167 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:30.384390 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:30.384391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:30.384392 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:54:30.384450 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:30.384453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.385216 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.385217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:30.386173 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:30.386176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.386779 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.386780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:30.386858 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:30.386879 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:54:30.387219 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:54:30.387228 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 04:54:30.387288 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 04:54:30.387355 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 04:54:30.387442 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:30.384058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.0.98.1:40913)
0928 04:54:30.384133 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:30.384135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:30.384145 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:54:30.384282 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:30.384283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:30.384305 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:30.384387 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:54:30.384516 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.385111 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.386405 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.386695 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.386757 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:30.387020 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:54:30.387137 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:54:30.387223 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:30.387289 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 04:54:30.387311 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 04:54:30.388804 8246 raft_consensus.cc:1261] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85 [term 1 FOLLOWER]: Refusing update from remote peer 59f9ec8981c54a28b7ea086466a70650: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:54:30.388936 8386 raft_consensus.cc:1261] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde [term 1 FOLLOWER]: Refusing update from remote peer 59f9ec8981c54a28b7ea086466a70650: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:54:30.389092 8506 consensus_queue.cc:1035] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:54:30.389209 8468 consensus_queue.cc:1035] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650 [LEADER]: Connected to new peer: Peer: permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:54:30.389882 8469 tablet_replica.cc:439] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85: TxnStatusTablet state changed. Reason: New leader 59f9ec8981c54a28b7ea086466a70650. Latest consensus state: current_term: 1 leader_uuid: "59f9ec8981c54a28b7ea086466a70650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } }
I20260928 04:54:30.389981 8469 tablet_replica.cc:442] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:54:30.389909 8484 tablet_replica.cc:439] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde: TxnStatusTablet state changed. Reason: New leader 59f9ec8981c54a28b7ea086466a70650. Latest consensus state: current_term: 1 leader_uuid: "59f9ec8981c54a28b7ea086466a70650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } }
I20260928 04:54:30.390008 8484 tablet_replica.cc:442] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:54:30.390717 8468 tablet_replica.cc:439] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59f9ec8981c54a28b7ea086466a70650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } }
I20260928 04:54:30.390810 8468 tablet_replica.cc:442] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:54:30.390865 8469 tablet_replica.cc:439] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59f9ec8981c54a28b7ea086466a70650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } }
I20260928 04:54:30.390954 8469 tablet_replica.cc:442] T bd8eeaf5c0a54adbb79735af42db007d P 0039d969e18240cca2fb882dcb844d85: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:54:30.392074 8468 tablet_replica.cc:439] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59f9ec8981c54a28b7ea086466a70650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } }
I20260928 04:54:30.392165 8468 tablet_replica.cc:442] T bd8eeaf5c0a54adbb79735af42db007d P 59f9ec8981c54a28b7ea086466a70650: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:54:30.392244 8484 tablet_replica.cc:439] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59f9ec8981c54a28b7ea086466a70650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } }
I20260928 04:54:30.392365 8484 tablet_replica.cc:442] T bd8eeaf5c0a54adbb79735af42db007d P 734f02ecf4374b989840a2f278efedde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:54:30.395623 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:30.392567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58018 (local address 127.0.98.2:34485)
0928 04:54:30.392656 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:30.392658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:30.392667 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:54:30.392708 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:30.392709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:30.392730 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:30.392807 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:54:30.392954 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.393713 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.394644 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.394953 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.395027 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:30.395187 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:54:30.395345 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:54:30.395425 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:30.395512 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 04:54:30.395537 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 04:54:30.401778 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:30.398882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40913 (local address 127.0.98.2:43045)
0928 04:54:30.399115 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:30.399125 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:30.399201 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:30.399328 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:30.399329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:30.399330 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:54:30.399405 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:30.399406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.400205 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.400206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:30.400909 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:30.400918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.401314 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.401315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:30.401376 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:30.401393 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:54:30.401590 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:54:30.401592 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:54:30.401642 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 04:54:30.401691 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":108,"threads_started":1}
I20260928 04:54:30.401752 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:30.398960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43045 (local address 127.0.98.1:40913)
0928 04:54:30.399044 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:30.399046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:30.399186 (+ 140us) server_negotiation.cc:407] Connection header received
0928 04:54:30.399257 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:30.399258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:30.399269 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:30.399325 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:54:30.399490 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.400105 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.401001 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.401197 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.401282 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:30.401486 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:54:30.401526 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:54:30.401588 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:30.401644 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 04:54:30.401669 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:54:30.404369 8386 raft_consensus.cc:1261] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde [term 1 FOLLOWER]: Refusing update from remote peer 0039d969e18240cca2fb882dcb844d85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:54:30.404436 8107 raft_consensus.cc:1261] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650 [term 1 FOLLOWER]: Refusing update from remote peer 0039d969e18240cca2fb882dcb844d85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:54:30.404639 8469 consensus_queue.cc:1035] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:54:30.404788 8495 consensus_queue.cc:1035] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:54:30.414135 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:30.411122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58971 (local address 127.0.98.2:34485)
0928 04:54:30.411165 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:30.411167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:30.411305 (+ 138us) server_negotiation.cc:407] Connection header received
0928 04:54:30.411338 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:30.411339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:30.411351 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:30.411408 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:54:30.411587 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.412063 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.413233 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.413558 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.413601 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:30.413862 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:54:30.413907 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:54:30.413960 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:30.414041 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 04:54:30.414063 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:54:30.414187 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:30.411061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34485 (local address 127.0.98.1:58971)
0928 04:54:30.411223 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:30.411236 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:30.411296 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:30.411458 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:30.411460 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:30.411463 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:54:30.411525 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:30.411526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.412142 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.412143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:30.413128 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:30.413131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.413658 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.413658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:30.413728 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:30.413748 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:54:30.413981 (+ 233us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:54:30.413983 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:54:30.414038 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 04:54:30.414090 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260928 04:54:30.425302 7949 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:54:30.425454 7949 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:54:30.426158 7949 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78: Sending DeleteTablet for 3 replicas of tablet 43084050970041528e1a23f6690f3b7b
I20260928 04:54:30.426433 8087 tablet_service.cc:1514] Processing DeleteTablet for tablet 43084050970041528e1a23f6690f3b7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:54:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56722
I20260928 04:54:30.426537 8227 tablet_service.cc:1514] Processing DeleteTablet for tablet 43084050970041528e1a23f6690f3b7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:54:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58010
I20260928 04:54:30.426560 8367 tablet_service.cc:1514] Processing DeleteTablet for tablet 43084050970041528e1a23f6690f3b7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:54:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41580
I20260928 04:54:30.426647 8488 tablet_replica.cc:330] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85: stopping tablet replica
I20260928 04:54:30.426673 8489 tablet_replica.cc:330] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650: stopping tablet replica
I20260928 04:54:30.426687 8487 tablet_replica.cc:330] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde: stopping tablet replica
I20260928 04:54:30.426707 8488 raft_consensus.cc:2229] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:54:30.426734 8489 raft_consensus.cc:2229] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:54:30.426751 8487 raft_consensus.cc:2229] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:54:30.426779 8489 raft_consensus.cc:2258] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:54:30.426800 8488 raft_consensus.cc:2258] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:54:30.426788 8487 raft_consensus.cc:2258] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:54:30.427220 8489 ts_tablet_manager.cc:1905] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:54:30.427496 8487 ts_tablet_manager.cc:1905] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:54:30.429222 8487 ts_tablet_manager.cc:1918] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:54:30.429246 8489 ts_tablet_manager.cc:1918] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:54:30.429296 8487 log.cc:1196] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/43084050970041528e1a23f6690f3b7b
I20260928 04:54:30.429318 8489 log.cc:1196] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/wal/wals/43084050970041528e1a23f6690f3b7b
I20260928 04:54:30.429569 8487 ts_tablet_manager.cc:1939] T 43084050970041528e1a23f6690f3b7b P 734f02ecf4374b989840a2f278efedde: Deleting consensus metadata
I20260928 04:54:30.429587 8489 ts_tablet_manager.cc:1939] T 43084050970041528e1a23f6690f3b7b P 59f9ec8981c54a28b7ea086466a70650: Deleting consensus metadata
I20260928 04:54:30.429879 8488 ts_tablet_manager.cc:1905] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:54:30.429998 7935 catalog_manager.cc:4841] TS 734f02ecf4374b989840a2f278efedde (127.0.98.3:42111): tablet 43084050970041528e1a23f6690f3b7b (table test-table [id=e70e58e7d1f24bfda0a5514e317549ae]) successfully deleted
I20260928 04:54:30.431586 8488 ts_tablet_manager.cc:1918] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:54:30.431653 8488 log.cc:1196] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/43084050970041528e1a23f6690f3b7b
I20260928 04:54:30.431881 8488 ts_tablet_manager.cc:1939] T 43084050970041528e1a23f6690f3b7b P 0039d969e18240cca2fb882dcb844d85: Deleting consensus metadata
I20260928 04:54:30.432158 7936 catalog_manager.cc:4841] TS 59f9ec8981c54a28b7ea086466a70650 (127.0.98.1:40913): tablet 43084050970041528e1a23f6690f3b7b (table test-table [id=e70e58e7d1f24bfda0a5514e317549ae]) successfully deleted
I20260928 04:54:30.432293 7937 catalog_manager.cc:4841] TS 0039d969e18240cca2fb882dcb844d85 (127.0.98.2:34485): tablet 43084050970041528e1a23f6690f3b7b (table test-table [id=e70e58e7d1f24bfda0a5514e317549ae]) successfully deleted
Sep 28 04:54:30 dist-test-slave-m39l krb5kdc[7905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 04:54:30 dist-test-slave-m39l krb5kdc[7905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:30 dist-test-slave-m39l krb5kdc[7905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 04:54:30.452540 7999 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:30.446459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44692 (local address 127.0.98.62:45175)
0928 04:54:30.446533 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:30.446535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:30.446718 (+ 183us) server_negotiation.cc:407] Connection header received
0928 04:54:30.446813 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:30.446814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:30.446827 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:30.446901 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:30.447558 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.448117 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.448959 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.449182 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.449284 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:30.452484 (+ 3200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44692: BlockingRecv error: recv got EOF from 127.0.0.1:44692 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 04:54:30 dist-test-slave-m39l krb5kdc[7905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:54:30.463868 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:30.453849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.0.98.62:45175)
0928 04:54:30.453898 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:30.453901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:30.454132 (+ 231us) server_negotiation.cc:407] Connection header received
0928 04:54:30.454205 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:30.454206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:30.454217 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:30.454299 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:30.455025 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.455861 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.456955 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.457285 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.457333 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:30.460493 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:30.460495 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:30.460495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:30.460505 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:30.462359 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:30.462873 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:30.462874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:30.462875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:30.462956 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:30.463193 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:30.463194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:30.463194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:30.463349 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:30.463445 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:30.463671 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 04:54:30.463766 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:54:30.466099 7949 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:54:30.466300 7949 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:54:30.468019 8087 tablet_service.cc:1467] Processing CreateTablet for tablet a172e847b34e464e85048f6a95281def (DEFAULT_TABLE table=test-table [id=9a3470ba6fa54fd69057a2fa4cad1515]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:54:30.468139 8227 tablet_service.cc:1467] Processing CreateTablet for tablet a172e847b34e464e85048f6a95281def (DEFAULT_TABLE table=test-table [id=9a3470ba6fa54fd69057a2fa4cad1515]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:54:30.468183 8087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a172e847b34e464e85048f6a95281def. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:30.468282 8227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a172e847b34e464e85048f6a95281def. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:30.469655 8461 tablet_bootstrap.cc:492] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650: Bootstrap starting.
I20260928 04:54:30.469807 8463 tablet_bootstrap.cc:492] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85: Bootstrap starting.
I20260928 04:54:30.470081 8461 tablet_bootstrap.cc:654] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:30.470289 8463 tablet_bootstrap.cc:654] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:30.470810 8367 tablet_service.cc:1467] Processing CreateTablet for tablet a172e847b34e464e85048f6a95281def (DEFAULT_TABLE table=test-table [id=9a3470ba6fa54fd69057a2fa4cad1515]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:54:30.470970 8367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a172e847b34e464e85048f6a95281def. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:30.471230 8461 tablet_bootstrap.cc:492] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650: No bootstrap required, opened a new log
I20260928 04:54:30.471279 8461 ts_tablet_manager.cc:1397] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:54:30.471441 8461 raft_consensus.cc:348] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.471510 8461 raft_consensus.cc:374] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:30.471530 8461 raft_consensus.cc:729] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f9ec8981c54a28b7ea086466a70650, State: Initialized, Role: FOLLOWER
I20260928 04:54:30.471573 8461 consensus_queue.cc:260] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.471655 8461 ts_tablet_manager.cc:1428] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:30.472489 8462 tablet_bootstrap.cc:492] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde: Bootstrap starting.
I20260928 04:54:30.472892 8462 tablet_bootstrap.cc:654] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:30.473717 8462 tablet_bootstrap.cc:492] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde: No bootstrap required, opened a new log
I20260928 04:54:30.473778 8462 ts_tablet_manager.cc:1397] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:30.473896 8462 raft_consensus.cc:348] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.473994 8462 raft_consensus.cc:374] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:30.474020 8462 raft_consensus.cc:729] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734f02ecf4374b989840a2f278efedde, State: Initialized, Role: FOLLOWER
I20260928 04:54:30.474074 8462 consensus_queue.cc:260] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.474181 8462 ts_tablet_manager.cc:1428] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:54:30.474880 8463 tablet_bootstrap.cc:492] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85: No bootstrap required, opened a new log
I20260928 04:54:30.474937 8463 ts_tablet_manager.cc:1397] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260928 04:54:30.475082 8463 raft_consensus.cc:348] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.475152 8463 raft_consensus.cc:374] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:30.475168 8463 raft_consensus.cc:729] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0039d969e18240cca2fb882dcb844d85, State: Initialized, Role: FOLLOWER
I20260928 04:54:30.475214 8463 consensus_queue.cc:260] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.475296 8463 ts_tablet_manager.cc:1428] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:30.478068 8513 raft_consensus.cc:482] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:54:30.478168 8513 raft_consensus.cc:504] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.478348 8513 leader_election.cc:288] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0039d969e18240cca2fb882dcb844d85 (127.0.98.2:34485), 734f02ecf4374b989840a2f278efedde (127.0.98.3:42111)
I20260928 04:54:30.478543 8386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a172e847b34e464e85048f6a95281def" candidate_uuid: "59f9ec8981c54a28b7ea086466a70650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734f02ecf4374b989840a2f278efedde" is_pre_election: true
I20260928 04:54:30.478641 8386 raft_consensus.cc:2454] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f9ec8981c54a28b7ea086466a70650 in term 0.
I20260928 04:54:30.478614 8246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a172e847b34e464e85048f6a95281def" candidate_uuid: "59f9ec8981c54a28b7ea086466a70650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0039d969e18240cca2fb882dcb844d85" is_pre_election: true
I20260928 04:54:30.478737 8246 raft_consensus.cc:2454] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f9ec8981c54a28b7ea086466a70650 in term 0.
I20260928 04:54:30.478802 8039 leader_election.cc:302] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f9ec8981c54a28b7ea086466a70650, 734f02ecf4374b989840a2f278efedde; no voters:
I20260928 04:54:30.479019 8513 raft_consensus.cc:2781] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:54:30.479055 8513 raft_consensus.cc:482] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:54:30.479074 8513 raft_consensus.cc:3037] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:30.480253 8513 raft_consensus.cc:504] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.480391 8513 leader_election.cc:288] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [CANDIDATE]: Term 1 election: Requested vote from peers 0039d969e18240cca2fb882dcb844d85 (127.0.98.2:34485), 734f02ecf4374b989840a2f278efedde (127.0.98.3:42111)
I20260928 04:54:30.480616 8386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a172e847b34e464e85048f6a95281def" candidate_uuid: "59f9ec8981c54a28b7ea086466a70650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734f02ecf4374b989840a2f278efedde"
I20260928 04:54:30.480687 8386 raft_consensus.cc:3037] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:30.480636 8246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a172e847b34e464e85048f6a95281def" candidate_uuid: "59f9ec8981c54a28b7ea086466a70650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0039d969e18240cca2fb882dcb844d85"
I20260928 04:54:30.480720 8246 raft_consensus.cc:3037] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:30.481488 8386 raft_consensus.cc:2454] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f9ec8981c54a28b7ea086466a70650 in term 1.
I20260928 04:54:30.481647 8039 leader_election.cc:302] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f9ec8981c54a28b7ea086466a70650, 734f02ecf4374b989840a2f278efedde; no voters:
I20260928 04:54:30.481709 8246 raft_consensus.cc:2454] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f9ec8981c54a28b7ea086466a70650 in term 1.
I20260928 04:54:30.481911 8513 raft_consensus.cc:2781] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:54:30.481959 8513 raft_consensus.cc:686] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [term 1 LEADER]: Becoming Leader. State: Replica: 59f9ec8981c54a28b7ea086466a70650, State: Running, Role: LEADER
I20260928 04:54:30.482004 8513 consensus_queue.cc:237] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.482688 7949 catalog_manager.cc:5495] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f9ec8981c54a28b7ea086466a70650 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "59f9ec8981c54a28b7ea086466a70650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } health_report { overall_health: UNKNOWN } } }
I20260928 04:54:30.490849 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:30.487252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.0.98.1:40913)
0928 04:54:30.487303 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:30.487305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:30.487393 (+ 88us) server_negotiation.cc:407] Connection header received
0928 04:54:30.487509 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:30.487510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:30.487522 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:30.487556 (+ 34us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:54:30.487769 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.488593 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.489855 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.490180 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.490232 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:30.490479 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:54:30.490566 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:54:30.490601 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:30.490728 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 04:54:30.490755 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:54:30.492169 8386 raft_consensus.cc:1261] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde [term 1 FOLLOWER]: Refusing update from remote peer 59f9ec8981c54a28b7ea086466a70650: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:54:30.492554 8513 consensus_queue.cc:1035] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [LEADER]: Connected to new peer: Peer: permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:54:30.493178 8246 raft_consensus.cc:1261] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85 [term 1 FOLLOWER]: Refusing update from remote peer 59f9ec8981c54a28b7ea086466a70650: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:54:30.493535 8513 consensus_queue.cc:1035] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:54:30.498395 7949 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:54:30.498560 7949 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:54:30.499281 7949 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78: Sending DeleteTablet for 3 replicas of tablet a172e847b34e464e85048f6a95281def
I20260928 04:54:30.499604 8367 tablet_service.cc:1514] Processing DeleteTablet for tablet a172e847b34e464e85048f6a95281def with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:54:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41580
I20260928 04:54:30.499714 8487 tablet_replica.cc:330] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde: stopping tablet replica
I20260928 04:54:30.499768 8487 raft_consensus.cc:2229] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:54:30.499816 8487 raft_consensus.cc:2258] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:54:30.500077 8087 tablet_service.cc:1514] Processing DeleteTablet for tablet a172e847b34e464e85048f6a95281def with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:54:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56722
I20260928 04:54:30.500177 8227 tablet_service.cc:1514] Processing DeleteTablet for tablet a172e847b34e464e85048f6a95281def with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:54:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58010
I20260928 04:54:30.500280 8489 tablet_replica.cc:330] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650: stopping tablet replica
I20260928 04:54:30.500250 7949 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 04:54:30.500337 8489 raft_consensus.cc:2229] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [term 1 LEADER]: Raft consensus shutting down.
W20260928 04:54:30.500380 7949 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:54:30.500418 8489 raft_consensus.cc:2258] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:54:30.500499 8488 tablet_replica.cc:330] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85: stopping tablet replica
I20260928 04:54:30.500556 8488 raft_consensus.cc:2229] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:54:30.500600 8488 raft_consensus.cc:2258] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:54:30.501098 8489 ts_tablet_manager.cc:1905] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:54:30.502820 8488 ts_tablet_manager.cc:1905] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:54:30.503204 8367 tablet_service.cc:1467] Processing CreateTablet for tablet 23dca895405b4391889b0748f577cdc7 (DEFAULT_TABLE table=test-table [id=5148fef817f243f9961824b88120be27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:54:30.503320 8367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23dca895405b4391889b0748f577cdc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:30.504364 8488 ts_tablet_manager.cc:1918] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:54:30.504415 8488 log.cc:1196] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/a172e847b34e464e85048f6a95281def
I20260928 04:54:30.504580 8462 tablet_bootstrap.cc:492] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde: Bootstrap starting.
I20260928 04:54:30.504628 8488 ts_tablet_manager.cc:1939] T a172e847b34e464e85048f6a95281def P 0039d969e18240cca2fb882dcb844d85: Deleting consensus metadata
I20260928 04:54:30.504863 8462 tablet_bootstrap.cc:654] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:30.505285 7937 catalog_manager.cc:4841] TS 0039d969e18240cca2fb882dcb844d85 (127.0.98.2:34485): tablet a172e847b34e464e85048f6a95281def (table test-table [id=9a3470ba6fa54fd69057a2fa4cad1515]) successfully deleted
I20260928 04:54:30.505384 8227 tablet_service.cc:1467] Processing CreateTablet for tablet 23dca895405b4391889b0748f577cdc7 (DEFAULT_TABLE table=test-table [id=5148fef817f243f9961824b88120be27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:54:30.505461 8087 tablet_service.cc:1467] Processing CreateTablet for tablet 23dca895405b4391889b0748f577cdc7 (DEFAULT_TABLE table=test-table [id=5148fef817f243f9961824b88120be27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:54:30.505483 8227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23dca895405b4391889b0748f577cdc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:30.505971 8489 ts_tablet_manager.cc:1918] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:54:30.506044 8489 log.cc:1196] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-0/wal/wals/a172e847b34e464e85048f6a95281def
I20260928 04:54:30.506258 8489 ts_tablet_manager.cc:1939] T a172e847b34e464e85048f6a95281def P 59f9ec8981c54a28b7ea086466a70650: Deleting consensus metadata
I20260928 04:54:30.506381 8087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23dca895405b4391889b0748f577cdc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:30.507625 8463 tablet_bootstrap.cc:492] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85: Bootstrap starting.
I20260928 04:54:30.507627 8461 tablet_bootstrap.cc:492] T 23dca895405b4391889b0748f577cdc7 P 59f9ec8981c54a28b7ea086466a70650: Bootstrap starting.
I20260928 04:54:30.508015 8463 tablet_bootstrap.cc:654] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:30.508049 8461 tablet_bootstrap.cc:654] T 23dca895405b4391889b0748f577cdc7 P 59f9ec8981c54a28b7ea086466a70650: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:30.508867 7936 catalog_manager.cc:4841] TS 59f9ec8981c54a28b7ea086466a70650 (127.0.98.1:40913): tablet a172e847b34e464e85048f6a95281def (table test-table [id=9a3470ba6fa54fd69057a2fa4cad1515]) successfully deleted
I20260928 04:54:30.508992 8487 ts_tablet_manager.cc:1905] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:54:30.510453 8487 ts_tablet_manager.cc:1918] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:54:30.510558 8487 log.cc:1196] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/a172e847b34e464e85048f6a95281def
I20260928 04:54:30.510793 8487 ts_tablet_manager.cc:1939] T a172e847b34e464e85048f6a95281def P 734f02ecf4374b989840a2f278efedde: Deleting consensus metadata
I20260928 04:54:30.510951 8462 tablet_bootstrap.cc:492] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde: No bootstrap required, opened a new log
I20260928 04:54:30.511009 8462 ts_tablet_manager.cc:1397] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 04:54:30.511127 7935 catalog_manager.cc:4841] TS 734f02ecf4374b989840a2f278efedde (127.0.98.3:42111): tablet a172e847b34e464e85048f6a95281def (table test-table [id=9a3470ba6fa54fd69057a2fa4cad1515]) successfully deleted
I20260928 04:54:30.511125 8462 raft_consensus.cc:348] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.511193 8462 raft_consensus.cc:374] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:30.511214 8462 raft_consensus.cc:729] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734f02ecf4374b989840a2f278efedde, State: Initialized, Role: FOLLOWER
I20260928 04:54:30.511260 8462 consensus_queue.cc:260] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.511343 8462 ts_tablet_manager.cc:1428] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:30.511794 8463 tablet_bootstrap.cc:492] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85: No bootstrap required, opened a new log
I20260928 04:54:30.511843 8463 ts_tablet_manager.cc:1397] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.000s
I20260928 04:54:30.511997 8463 raft_consensus.cc:348] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.512084 8463 raft_consensus.cc:374] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:30.512104 8463 raft_consensus.cc:729] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0039d969e18240cca2fb882dcb844d85, State: Initialized, Role: FOLLOWER
I20260928 04:54:30.512141 8463 consensus_queue.cc:260] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.512217 8463 ts_tablet_manager.cc:1428] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:30.515007 8461 tablet_bootstrap.cc:492] T 23dca895405b4391889b0748f577cdc7 P 59f9ec8981c54a28b7ea086466a70650: No bootstrap required, opened a new log
I20260928 04:54:30.515079 8461 ts_tablet_manager.cc:1397] T 23dca895405b4391889b0748f577cdc7 P 59f9ec8981c54a28b7ea086466a70650: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.000s
I20260928 04:54:30.515235 8461 raft_consensus.cc:348] T 23dca895405b4391889b0748f577cdc7 P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.515322 8461 raft_consensus.cc:374] T 23dca895405b4391889b0748f577cdc7 P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:30.515340 8461 raft_consensus.cc:729] T 23dca895405b4391889b0748f577cdc7 P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f9ec8981c54a28b7ea086466a70650, State: Initialized, Role: FOLLOWER
I20260928 04:54:30.515384 8461 consensus_queue.cc:260] T 23dca895405b4391889b0748f577cdc7 P 59f9ec8981c54a28b7ea086466a70650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.515470 8461 ts_tablet_manager.cc:1428] T 23dca895405b4391889b0748f577cdc7 P 59f9ec8981c54a28b7ea086466a70650: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:30.598745 8495 raft_consensus.cc:482] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:54:30.598943 8495 raft_consensus.cc:504] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.599143 8495 leader_election.cc:288] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f9ec8981c54a28b7ea086466a70650 (127.0.98.1:40913), 734f02ecf4374b989840a2f278efedde (127.0.98.3:42111)
I20260928 04:54:30.599400 8107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dca895405b4391889b0748f577cdc7" candidate_uuid: "0039d969e18240cca2fb882dcb844d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f9ec8981c54a28b7ea086466a70650" is_pre_election: true
I20260928 04:54:30.599397 8386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dca895405b4391889b0748f577cdc7" candidate_uuid: "0039d969e18240cca2fb882dcb844d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734f02ecf4374b989840a2f278efedde" is_pre_election: true
I20260928 04:54:30.599512 8107 raft_consensus.cc:2454] T 23dca895405b4391889b0748f577cdc7 P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0039d969e18240cca2fb882dcb844d85 in term 0.
I20260928 04:54:30.599545 8386 raft_consensus.cc:2454] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0039d969e18240cca2fb882dcb844d85 in term 0.
I20260928 04:54:30.599699 8180 leader_election.cc:302] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0039d969e18240cca2fb882dcb844d85, 59f9ec8981c54a28b7ea086466a70650; no voters:
I20260928 04:54:30.599825 8495 raft_consensus.cc:2781] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:54:30.599879 8495 raft_consensus.cc:482] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:54:30.599911 8495 raft_consensus.cc:3037] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:30.600782 8495 raft_consensus.cc:504] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.600958 8495 leader_election.cc:288] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [CANDIDATE]: Term 1 election: Requested vote from peers 59f9ec8981c54a28b7ea086466a70650 (127.0.98.1:40913), 734f02ecf4374b989840a2f278efedde (127.0.98.3:42111)
I20260928 04:54:30.601125 8107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dca895405b4391889b0748f577cdc7" candidate_uuid: "0039d969e18240cca2fb882dcb844d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f9ec8981c54a28b7ea086466a70650"
I20260928 04:54:30.601207 8107 raft_consensus.cc:3037] T 23dca895405b4391889b0748f577cdc7 P 59f9ec8981c54a28b7ea086466a70650 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:30.601215 8386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dca895405b4391889b0748f577cdc7" candidate_uuid: "0039d969e18240cca2fb882dcb844d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734f02ecf4374b989840a2f278efedde"
I20260928 04:54:30.601310 8386 raft_consensus.cc:3037] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:30.601796 8107 raft_consensus.cc:2454] T 23dca895405b4391889b0748f577cdc7 P 59f9ec8981c54a28b7ea086466a70650 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0039d969e18240cca2fb882dcb844d85 in term 1.
I20260928 04:54:30.601994 8180 leader_election.cc:302] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0039d969e18240cca2fb882dcb844d85, 59f9ec8981c54a28b7ea086466a70650; no voters:
I20260928 04:54:30.602120 8495 raft_consensus.cc:2781] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:54:30.602173 8495 raft_consensus.cc:686] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [term 1 LEADER]: Becoming Leader. State: Replica: 0039d969e18240cca2fb882dcb844d85, State: Running, Role: LEADER
I20260928 04:54:30.602214 8386 raft_consensus.cc:2454] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0039d969e18240cca2fb882dcb844d85 in term 1.
I20260928 04:54:30.602226 8495 consensus_queue.cc:237] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260928 04:54:30.602900 7950 catalog_manager.cc:5495] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0039d969e18240cca2fb882dcb844d85 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0039d969e18240cca2fb882dcb844d85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0039d969e18240cca2fb882dcb844d85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } health_report { overall_health: UNKNOWN } } }
I20260928 04:54:30.633776 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:30.630724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58020 (local address 127.0.98.2:34485)
0928 04:54:30.630768 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:30.630770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:30.630835 (+ 65us) server_negotiation.cc:407] Connection header received
0928 04:54:30.630941 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:30.630941 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:30.630950 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:30.630995 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:54:30.631128 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.631594 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.632824 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:30.633064 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:30.633130 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:30.633408 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:54:30.633444 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:54:30.633503 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:30.633638 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 04:54:30.633658 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:54:30.636507 8107 raft_consensus.cc:1261] T 23dca895405b4391889b0748f577cdc7 P 59f9ec8981c54a28b7ea086466a70650 [term 1 FOLLOWER]: Refusing update from remote peer 0039d969e18240cca2fb882dcb844d85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:54:30.636499 8387 raft_consensus.cc:1261] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde [term 1 FOLLOWER]: Refusing update from remote peer 0039d969e18240cca2fb882dcb844d85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:54:30.636737 8495 consensus_queue.cc:1035] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f9ec8981c54a28b7ea086466a70650" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:54:30.636873 8469 consensus_queue.cc:1035] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "734f02ecf4374b989840a2f278efedde" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:54:30.647842 7950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:54:30.647970 7950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:54:30.648625 7950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5821ada73b2747f98afff71a9c0a9e78: Sending DeleteTablet for 3 replicas of tablet 23dca895405b4391889b0748f577cdc7
I20260928 04:54:30.648905 8367 tablet_service.cc:1514] Processing DeleteTablet for tablet 23dca895405b4391889b0748f577cdc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:54:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41580
I20260928 04:54:30.649022 8227 tablet_service.cc:1514] Processing DeleteTablet for tablet 23dca895405b4391889b0748f577cdc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:54:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58010
I20260928 04:54:30.649041 8487 tablet_replica.cc:330] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde: stopping tablet replica
I20260928 04:54:30.649111 8488 tablet_replica.cc:330] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85: stopping tablet replica
I20260928 04:54:30.649111 8487 raft_consensus.cc:2229] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:54:30.649148 8487 raft_consensus.cc:2258] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:54:30.649156 8488 raft_consensus.cc:2229] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:54:30.649215 8488 raft_consensus.cc:2258] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:54:30.649358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 8022
I20260928 04:54:30.649421 8087 tablet_service.cc:1514] Processing DeleteTablet for tablet 23dca895405b4391889b0748f577cdc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:54:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56722
I20260928 04:54:30.651700 8487 ts_tablet_manager.cc:1905] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:54:30.653244 8487 ts_tablet_manager.cc:1918] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:54:30.653339 8487 log.cc:1196] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/23dca895405b4391889b0748f577cdc7
I20260928 04:54:30.653571 8487 ts_tablet_manager.cc:1939] T 23dca895405b4391889b0748f577cdc7 P 734f02ecf4374b989840a2f278efedde: Deleting consensus metadata
I20260928 04:54:30.653975 7935 catalog_manager.cc:4841] TS 734f02ecf4374b989840a2f278efedde (127.0.98.3:42111): tablet 23dca895405b4391889b0748f577cdc7 (table test-table [id=5148fef817f243f9961824b88120be27]) successfully deleted
I20260928 04:54:30.655223 8488 ts_tablet_manager.cc:1905] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:54:30.656684 8488 ts_tablet_manager.cc:1918] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:54:30.656747 8488 log.cc:1196] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/23dca895405b4391889b0748f577cdc7
I20260928 04:54:30.656996 8488 ts_tablet_manager.cc:1939] T 23dca895405b4391889b0748f577cdc7 P 0039d969e18240cca2fb882dcb844d85: Deleting consensus metadata
I20260928 04:54:30.657387 7937 catalog_manager.cc:4841] TS 0039d969e18240cca2fb882dcb844d85 (127.0.98.2:34485): tablet 23dca895405b4391889b0748f577cdc7 (table test-table [id=5148fef817f243f9961824b88120be27]) successfully deleted
W20260928 04:54:30.657855 7936 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40913 (error 108)
I20260928 04:54:30.658056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 8162
I20260928 04:54:30.659608 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:30.659327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40913 (local address 127.0.0.1:56746)
0928 04:54:30.659511 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:30.659569 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":98,"threads_started":1}
W20260928 04:54:30.659835 7936 catalog_manager.cc:4584] TS 59f9ec8981c54a28b7ea086466a70650 (127.0.98.1:40913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 23dca895405b4391889b0748f577cdc7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40913: connect: Connection refused (error 111)
I20260928 04:54:30.664650 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 8302
I20260928 04:54:30.671389 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 7918
2026-09-28T04:54:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2759 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:54:30 dist-test-slave-m39l krb5kdc[8553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:54:30 dist-test-slave-m39l krb5kdc[8553](info): set up 2 sockets
Sep 28 04:54:30 dist-test-slave-m39l krb5kdc[8553](info): commencing operation
krb5kdc: starting...
W20260928 04:54:31.723119 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:54:31 dist-test-slave-m39l krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:54:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:54:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:54:31.853981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41237
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:54:31.937903 8566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:31.938179 8566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:31.938221 8566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:31.939553 8566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:54:31.939628 8566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:31.939652 8566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:31.939682 8566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:54:31.939714 8566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:54:31.941327 8566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:31.941625 8566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:31.941875 8566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:31.947211 8574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:31.947237 8571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:31.947237 8572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:31.947288 8566 server_base.cc:1034] running on GCE node
I20260928 04:54:31.947767 8566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:31.948129 8566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:31.949303 8566 hybrid_clock.cc:648] HybridClock initialized: now 1790571271949279 us; error 42 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T04:54:31Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1279 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:54:32 dist-test-slave-m39l krb5kdc[8579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:54:32 dist-test-slave-m39l krb5kdc[8579](info): set up 2 sockets
Sep 28 04:54:32 dist-test-slave-m39l krb5kdc[8579](info): commencing operation
krb5kdc: starting...
W20260928 04:54:33.043180 392 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:54:33 dist-test-slave-m39l krb5kdc[8579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:54:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:54:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:54:33.170270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38585
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:54:33.256934 8592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:33.257203 8592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:33.257231 8592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:33.258773 8592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:54:33.258828 8592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:33.258843 8592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:33.258857 8592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:54:33.258868 8592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:54:33.260828 8592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:33.261160 8592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:33.261442 8592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:33.266875 8597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:33.266995 8592 server_base.cc:1034] running on GCE node
W20260928 04:54:33.266878 8600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:33.266954 8598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:33.267371 8592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:33.267745 8592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:33.268923 8592 hybrid_clock.cc:648] HybridClock initialized: now 1790571273268913 us; error 47 us; skew 500 ppm
Sep 28 04:54:33 dist-test-slave-m39l krb5kdc[8579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:54:33.278702 8592 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:54:33.279332 8592 webserver.cc:460] Webserver started at http://127.0.98.62:42713/ using document root <none> and password file <none>
I20260928 04:54:33.279690 8592 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:33.279773 8592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:33.279947 8592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:33.281132 8592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "23e946c774774d81bf04d38e4d5af5a1"
format_stamp: "Formatted at 2026-09-28 04:54:33 on dist-test-slave-m39l"
server_key: "b5e1af38db94fb1ee5d01bdc00f22464"
server_key_iv: "4d697c05b0c995fefdb0fc7bd40f3454"
server_key_version: "encryptionkey@0"
I20260928 04:54:33.281728 8592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "23e946c774774d81bf04d38e4d5af5a1"
format_stamp: "Formatted at 2026-09-28 04:54:33 on dist-test-slave-m39l"
server_key: "b5e1af38db94fb1ee5d01bdc00f22464"
server_key_iv: "4d697c05b0c995fefdb0fc7bd40f3454"
server_key_version: "encryptionkey@0"
I20260928 04:54:33.283515 8592 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 04:54:33.284457 8607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:33.284718 8592 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:33.284865 8592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "23e946c774774d81bf04d38e4d5af5a1"
format_stamp: "Formatted at 2026-09-28 04:54:33 on dist-test-slave-m39l"
server_key: "b5e1af38db94fb1ee5d01bdc00f22464"
server_key_iv: "4d697c05b0c995fefdb0fc7bd40f3454"
server_key_version: "encryptionkey@0"
I20260928 04:54:33.284963 8592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:33.297230 8592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:33.299474 8592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:33.299661 8592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:33.304104 8592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38585
I20260928 04:54:33.304140 8659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38585 every 8 connection(s)
I20260928 04:54:33.304697 8592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:54:33.305555 8660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:33.306259 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 8592
I20260928 04:54:33.306439 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:54:33.306691 392 external_mini_cluster.cc:1428] Setting key 9fcb8512f1bed134cffa31f62ad80e4e
I20260928 04:54:33.308918 8660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1: Bootstrap starting.
I20260928 04:54:33.309677 8660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:33.310002 8660 log.cc:824] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1: Log is configured to *not* fsync() on all Append() calls
I20260928 04:54:33.310807 8660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1: No bootstrap required, opened a new log
I20260928 04:54:33.313235 8660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e946c774774d81bf04d38e4d5af5a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38585 } }
I20260928 04:54:33.313397 8660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:33.313421 8660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23e946c774774d81bf04d38e4d5af5a1, State: Initialized, Role: FOLLOWER
I20260928 04:54:33.313577 8660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e946c774774d81bf04d38e4d5af5a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38585 } }
I20260928 04:54:33.313669 8660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:54:33.313720 8660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:54:33.313773 8660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 04:54:33 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:54:33.314544 8660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e946c774774d81bf04d38e4d5af5a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38585 } }
I20260928 04:54:33.314674 8660 leader_election.cc:302] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23e946c774774d81bf04d38e4d5af5a1; no voters:
I20260928 04:54:33.314858 8660 leader_election.cc:288] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:54:33.314905 8665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:54:33.315029 8665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [term 1 LEADER]: Becoming Leader. State: Replica: 23e946c774774d81bf04d38e4d5af5a1, State: Running, Role: LEADER
I20260928 04:54:33.315121 8665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e946c774774d81bf04d38e4d5af5a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38585 } }
I20260928 04:54:33.315145 8660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:54:33.315496 8665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23e946c774774d81bf04d38e4d5af5a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e946c774774d81bf04d38e4d5af5a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38585 } } }
I20260928 04:54:33.315574 8665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [sys.catalog]: This master's current role is: LEADER
I20260928 04:54:33.315635 8667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23e946c774774d81bf04d38e4d5af5a1. Latest consensus state: current_term: 1 leader_uuid: "23e946c774774d81bf04d38e4d5af5a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e946c774774d81bf04d38e4d5af5a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38585 } } }
I20260928 04:54:33.315704 8667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [sys.catalog]: This master's current role is: LEADER
I20260928 04:54:33.315924 8669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:54:33.316457 8669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:54:33.318135 8669 catalog_manager.cc:1347] Generated new cluster ID: 3a7a5d81a58b4e0d8ddcb6f31ea41110
I20260928 04:54:33.318183 8669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:54:33.319262 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:33.307289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57948 (local address 127.0.98.62:38585)
0928 04:54:33.307498 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:33.307509 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:33.307543 (+ 34us) server_negotiation.cc:407] Connection header received
0928 04:54:33.308127 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:33.308137 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:33.308209 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:33.308372 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:33.309058 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:33.310461 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:33.311326 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:33.311979 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:33.312127 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:33.314810 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:33.314815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:33.314815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:33.314827 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:33.317436 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:33.317961 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:33.317965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:33.317965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:33.318110 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:33.318312 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:33.318313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:33.318313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:33.318642 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:33.318811 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:33.318942 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 04:54:33.319083 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20260928 04:54:33.330452 8669 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:54:33.331081 8669 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:54:33.341305 8669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1: Generated new TSK 0
I20260928 04:54:33.341535 8669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:54:33.379531 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38585
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:54:33.466146 8686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:33.466380 8686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:33.466405 8686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:33.467643 8686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:33.467691 8686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:33.467720 8686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:54:33.469255 8686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:33.469471 8686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:33.469659 8686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:33.474982 8691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:33.474983 8686 server_base.cc:1034] running on GCE node
W20260928 04:54:33.474997 8694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:33.474999 8692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:33.475709 8686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:33.476018 8686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:33.477185 8686 hybrid_clock.cc:648] HybridClock initialized: now 1790571273477165 us; error 31 us; skew 500 ppm
Sep 28 04:54:33 dist-test-slave-m39l krb5kdc[8579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:54:33.479929 8686 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:54:33.480437 8686 webserver.cc:460] Webserver started at http://127.0.98.1:37225/ using document root <none> and password file <none>
I20260928 04:54:33.480675 8686 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:33.480746 8686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:33.480908 8686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:33.482048 8686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "0ebe5334907f417dbaf2e23b10659a6b"
format_stamp: "Formatted at 2026-09-28 04:54:33 on dist-test-slave-m39l"
server_key: "981a6744a3daa0ac4b1431a9c8c07a7c"
server_key_iv: "0de42389ca63ac0fc5a410015b594363"
server_key_version: "encryptionkey@0"
I20260928 04:54:33.482434 8686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0ebe5334907f417dbaf2e23b10659a6b"
format_stamp: "Formatted at 2026-09-28 04:54:33 on dist-test-slave-m39l"
server_key: "981a6744a3daa0ac4b1431a9c8c07a7c"
server_key_iv: "0de42389ca63ac0fc5a410015b594363"
server_key_version: "encryptionkey@0"
I20260928 04:54:33.483883 8686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:33.484858 8701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:33.485255 8686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:33.485443 8686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "0ebe5334907f417dbaf2e23b10659a6b"
format_stamp: "Formatted at 2026-09-28 04:54:33 on dist-test-slave-m39l"
server_key: "981a6744a3daa0ac4b1431a9c8c07a7c"
server_key_iv: "0de42389ca63ac0fc5a410015b594363"
server_key_version: "encryptionkey@0"
I20260928 04:54:33.485534 8686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:33.508251 8686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:33.511061 8686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:33.511260 8686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:33.511538 8686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:54:33.511884 8686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:54:33.511940 8686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:33.511984 8686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:54:33.512048 8686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:33.518599 8686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42459
I20260928 04:54:33.518640 8814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42459 every 8 connection(s)
I20260928 04:54:33.519001 8686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:54:33.525216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 8686
I20260928 04:54:33.525358 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:54:33.525575 392 external_mini_cluster.cc:1428] Setting key b2304d6e89f08a86613e1b83e2ea5056
Sep 28 04:54:33 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:54:33.530932 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:33.520167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46281 (local address 127.0.98.62:38585)
0928 04:54:33.520238 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:33.520240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:33.520324 (+ 84us) server_negotiation.cc:407] Connection header received
0928 04:54:33.521307 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:33.521308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:33.521327 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:33.521397 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:33.522110 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:33.522736 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:33.524130 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:33.524363 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:33.524431 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:33.527408 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:33.527409 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:33.527410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:33.527420 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:33.529049 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:33.529595 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:33.529597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:33.529597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:33.529682 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:33.529955 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:33.529956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:33.529956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:33.530124 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:33.530199 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:33.530644 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 04:54:33.530761 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 04:54:33.531090 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:33.520009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:46281)
0928 04:54:33.520178 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:33.520224 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:33.521183 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:33.521459 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:33.521462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:33.521745 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:33.521974 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:33.521979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:33.522872 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:33.522873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:33.524011 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:33.524014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:33.524458 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:33.524459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:33.524577 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:33.525171 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:33.525186 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:33.527269 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:33.529157 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:33.529169 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:33.529172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:33.529486 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:33.529762 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:33.529763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:33.529763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:33.529880 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:33.530219 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:33.530227 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:33.530382 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 04:54:33.530615 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 04:54:33.530844 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260928 04:54:33.531916 8815 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38585
I20260928 04:54:33.532066 8815 heartbeater.cc:464] Registering TS with master...
I20260928 04:54:33.532270 8815 heartbeater.cc:511] Master 127.0.98.62:38585 requested a full tablet report, sending...
I20260928 04:54:33.532971 8624 ts_manager.cc:194] Registered new tserver with Master: 0ebe5334907f417dbaf2e23b10659a6b (127.0.98.1:42459)
I20260928 04:54:33.533833 8624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46281
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:54:33.557821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38585
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:54:33.641963 8820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:33.642233 8820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:33.642277 8820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:33.643689 8820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:33.643762 8820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:33.643810 8820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:54:33.645475 8820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:33.645781 8820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:33.646032 8820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:33.651672 8825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:33.651949 8820 server_base.cc:1034] running on GCE node
W20260928 04:54:33.651670 8828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:33.651670 8826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:33.652361 8820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:33.652613 8820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:33.653805 8820 hybrid_clock.cc:648] HybridClock initialized: now 1790571273653771 us; error 48 us; skew 500 ppm
Sep 28 04:54:33 dist-test-slave-m39l krb5kdc[8579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:54:33.656539 8820 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:54:33.657030 8820 webserver.cc:460] Webserver started at http://127.0.98.2:36963/ using document root <none> and password file <none>
I20260928 04:54:33.657270 8820 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:33.657341 8820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:33.657550 8820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:33.658743 8820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "0e83419bc72749ecb97c941f03e4efc4"
format_stamp: "Formatted at 2026-09-28 04:54:33 on dist-test-slave-m39l"
server_key: "f21041281c760e22b57730d5fd59c873"
server_key_iv: "6259a549153bbe436e9f1c6d08e82ecc"
server_key_version: "encryptionkey@0"
I20260928 04:54:33.659139 8820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0e83419bc72749ecb97c941f03e4efc4"
format_stamp: "Formatted at 2026-09-28 04:54:33 on dist-test-slave-m39l"
server_key: "f21041281c760e22b57730d5fd59c873"
server_key_iv: "6259a549153bbe436e9f1c6d08e82ecc"
server_key_version: "encryptionkey@0"
I20260928 04:54:33.660579 8820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:54:33.661386 8835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:33.661624 8820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:33.661763 8820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "0e83419bc72749ecb97c941f03e4efc4"
format_stamp: "Formatted at 2026-09-28 04:54:33 on dist-test-slave-m39l"
server_key: "f21041281c760e22b57730d5fd59c873"
server_key_iv: "6259a549153bbe436e9f1c6d08e82ecc"
server_key_version: "encryptionkey@0"
I20260928 04:54:33.661844 8820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:33.681226 8820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:33.683763 8820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:33.683960 8820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:33.684237 8820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:54:33.684592 8820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:54:33.684648 8820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:33.684690 8820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:54:33.684719 8820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:33.691378 8820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43079
I20260928 04:54:33.691464 8948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43079 every 8 connection(s)
I20260928 04:54:33.691816 8820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:54:33.693979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 8820
I20260928 04:54:33.694109 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:54:33.694387 392 external_mini_cluster.cc:1428] Setting key d83a6b02365c24089f5d1affd773e259
Sep 28 04:54:33 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 04:54:33.704849 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:33.692706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60919 (local address 127.0.98.62:38585)
0928 04:54:33.692767 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:33.692769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:33.692983 (+ 214us) server_negotiation.cc:407] Connection header received
0928 04:54:33.693969 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:33.693971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:33.693984 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:33.694060 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:33.695055 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:33.695692 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:33.697604 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:33.697844 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:33.697943 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:33.700926 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:33.700928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:33.700928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:33.700936 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:33.702749 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:33.703477 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:33.703478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:33.703479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:33.703548 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:33.703902 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:33.703902 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:33.703903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:33.704035 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:33.704095 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:33.704665 (+ 570us) server_negotiation.cc:299] Negotiation successful
0928 04:54:33.704772 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 04:54:33.704996 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:33.692646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:60919)
0928 04:54:33.692845 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:33.692883 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:33.693832 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:33.694172 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:33.694177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:33.694476 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:33.694900 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:33.694907 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:33.695886 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:33.695888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:33.697485 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:33.697488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:33.697963 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:33.697964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:33.698114 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:33.698963 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:33.698973 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:33.700764 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:33.702856 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:33.702876 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:33.702879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:33.703250 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:33.703641 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:33.703643 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:33.703643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:33.703816 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:33.704109 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:33.704117 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:33.704317 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 04:54:33.704447 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 04:54:33.704662 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":66,"threads_started":1}
I20260928 04:54:33.705786 8949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38585
I20260928 04:54:33.705928 8949 heartbeater.cc:464] Registering TS with master...
I20260928 04:54:33.706190 8949 heartbeater.cc:511] Master 127.0.98.62:38585 requested a full tablet report, sending...
I20260928 04:54:33.706744 8624 ts_manager.cc:194] Registered new tserver with Master: 0e83419bc72749ecb97c941f03e4efc4 (127.0.98.2:43079)
I20260928 04:54:33.707201 8624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:54:33.726671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38585
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:54:33.814252 8954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:33.814517 8954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:33.814584 8954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:33.816155 8954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:33.816211 8954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:33.816242 8954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:54:33.818389 8954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:33.818801 8954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:33.819060 8954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:33.825258 8962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:33.825269 8959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:33.825379 8960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:33.825531 8954 server_base.cc:1034] running on GCE node
I20260928 04:54:33.825843 8954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:33.826123 8954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:33.827296 8954 hybrid_clock.cc:648] HybridClock initialized: now 1790571273827251 us; error 87 us; skew 500 ppm
Sep 28 04:54:33 dist-test-slave-m39l krb5kdc[8579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:54:33.830652 8954 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:54:33.831199 8954 webserver.cc:460] Webserver started at http://127.0.98.3:38835/ using document root <none> and password file <none>
I20260928 04:54:33.831468 8954 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:33.831552 8954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:33.831728 8954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:33.833106 8954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "da17231974834159b0421cc278683811"
format_stamp: "Formatted at 2026-09-28 04:54:33 on dist-test-slave-m39l"
server_key: "ea24038770eb7d45efb79f8b3d1ec035"
server_key_iv: "b3225c8a9b7070cfdb2d7adb6aec608f"
server_key_version: "encryptionkey@0"
I20260928 04:54:33.833535 8954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "da17231974834159b0421cc278683811"
format_stamp: "Formatted at 2026-09-28 04:54:33 on dist-test-slave-m39l"
server_key: "ea24038770eb7d45efb79f8b3d1ec035"
server_key_iv: "b3225c8a9b7070cfdb2d7adb6aec608f"
server_key_version: "encryptionkey@0"
I20260928 04:54:33.835086 8954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 04:54:33.836024 8969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:33.836247 8954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:33.836369 8954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "da17231974834159b0421cc278683811"
format_stamp: "Formatted at 2026-09-28 04:54:33 on dist-test-slave-m39l"
server_key: "ea24038770eb7d45efb79f8b3d1ec035"
server_key_iv: "b3225c8a9b7070cfdb2d7adb6aec608f"
server_key_version: "encryptionkey@0"
I20260928 04:54:33.836452 8954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:33.871176 8954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:33.874049 8954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:33.874255 8954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:33.874607 8954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:54:33.874975 8954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:54:33.875043 8954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:33.875092 8954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:54:33.875120 8954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:33.881721 8954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45611
I20260928 04:54:33.881779 9082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45611 every 8 connection(s)
I20260928 04:54:33.882177 8954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:54:33.882597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 8954
I20260928 04:54:33.882701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:54:33.882902 392 external_mini_cluster.cc:1428] Setting key c00e29ad5ac1576fc59db5a11734ea1f
Sep 28 04:54:33 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:54:33.893523 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:33.883382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42785 (local address 127.0.98.62:38585)
0928 04:54:33.883435 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:33.883438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:33.883571 (+ 133us) server_negotiation.cc:407] Connection header received
0928 04:54:33.884612 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:33.884613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:33.884627 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:33.884685 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:33.885392 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:33.885907 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:33.887043 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:33.887252 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:33.887347 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:33.890662 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:33.890664 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:33.890664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:33.890674 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:33.892092 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:33.892585 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:33.892586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:33.892586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:33.892644 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:33.892925 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:33.892925 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:33.892926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:33.893042 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:33.893097 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:33.893377 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 04:54:33.893448 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 04:54:33.893784 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:33.883110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.3:42785)
0928 04:54:33.883364 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:33.883467 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:33.884480 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:33.884771 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:33.884777 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:33.885080 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:33.885288 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:33.885293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:33.886015 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:33.886017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:33.886936 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:33.886938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:33.887362 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:33.887363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:33.887499 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:33.888165 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:33.888174 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:33.890483 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:33.892196 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:33.892206 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:33.892208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:33.892505 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:33.892711 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:33.892711 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:33.892712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:33.892862 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:33.893101 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:33.893106 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:33.893246 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 04:54:33.893341 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 04:54:33.893495 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":108,"threads_started":1}
I20260928 04:54:33.894583 9083 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38585
I20260928 04:54:33.894721 9083 heartbeater.cc:464] Registering TS with master...
I20260928 04:54:33.895004 9083 heartbeater.cc:511] Master 127.0.98.62:38585 requested a full tablet report, sending...
I20260928 04:54:33.895429 8623 ts_manager.cc:194] Registered new tserver with Master: da17231974834159b0421cc278683811 (127.0.98.3:45611)
I20260928 04:54:33.896024 8623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42785
I20260928 04:54:33.906148 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 04:54:33.999599 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 8686
I20260928 04:54:34.004438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 8820
I20260928 04:54:34.009425 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 8954
I20260928 04:54:34.014367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 8592
I20260928 04:54:34.020274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38585
--webserver_interface=127.0.98.62
--webserver_port=42713
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:54:34.111578 9092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:34.111830 9092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:34.111860 9092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:34.113297 9092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:54:34.113361 9092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:34.113377 9092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:34.113390 9092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:54:34.113404 9092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:54:34.115233 9092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:34.115492 9092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:34.115685 9092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:34.121163 9097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:34.121163 9098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:34.121163 9100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:34.121205 9092 server_base.cc:1034] running on GCE node
I20260928 04:54:34.121711 9092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:34.122072 9092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:34.123257 9092 hybrid_clock.cc:648] HybridClock initialized: now 1790571274123235 us; error 42 us; skew 500 ppm
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:54:34.126231 9092 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:54:34.126842 9092 webserver.cc:460] Webserver started at http://127.0.98.62:42713/ using document root <none> and password file <none>
I20260928 04:54:34.127121 9092 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:34.127224 9092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:34.128909 9092 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:54:34.129762 9107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:34.130021 9092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:34.130138 9092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "23e946c774774d81bf04d38e4d5af5a1"
format_stamp: "Formatted at 2026-09-28 04:54:33 on dist-test-slave-m39l"
server_key: "b5e1af38db94fb1ee5d01bdc00f22464"
server_key_iv: "4d697c05b0c995fefdb0fc7bd40f3454"
server_key_version: "encryptionkey@0"
I20260928 04:54:34.130649 9092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:34.142719 9092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:34.145306 9092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:34.145511 9092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:34.149883 9092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38585
I20260928 04:54:34.149986 9159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38585 every 8 connection(s)
I20260928 04:54:34.150602 9092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:54:34.152199 9160 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:54:34.153017 9160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1: Bootstrap starting.
I20260928 04:54:34.155807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 9092
I20260928 04:54:34.156117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42459
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38585
--webserver_port=37225
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:54:34.161959 9160 log.cc:824] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1: Log is configured to *not* fsync() on all Append() calls
I20260928 04:54:34.163831 9160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:54:34.164149 9160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1: Bootstrap complete.
I20260928 04:54:34.166468 9160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e946c774774d81bf04d38e4d5af5a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38585 } }
I20260928 04:54:34.166826 9160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23e946c774774d81bf04d38e4d5af5a1, State: Initialized, Role: FOLLOWER
I20260928 04:54:34.167017 9160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e946c774774d81bf04d38e4d5af5a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38585 } }
I20260928 04:54:34.167095 9160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:54:34.167146 9160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:54:34.167199 9160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [term 1 FOLLOWER]: Advancing to term 2
I20260928 04:54:34.168243 9160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e946c774774d81bf04d38e4d5af5a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38585 } }
I20260928 04:54:34.168375 9160 leader_election.cc:302] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23e946c774774d81bf04d38e4d5af5a1; no voters:
I20260928 04:54:34.168578 9160 leader_election.cc:288] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 04:54:34.168648 9164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 04:54:34.168812 9164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [term 2 LEADER]: Becoming Leader. State: Replica: 23e946c774774d81bf04d38e4d5af5a1, State: Running, Role: LEADER
I20260928 04:54:34.168869 9160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:54:34.168910 9164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e946c774774d81bf04d38e4d5af5a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38585 } }
I20260928 04:54:34.169178 9164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23e946c774774d81bf04d38e4d5af5a1. Latest consensus state: current_term: 2 leader_uuid: "23e946c774774d81bf04d38e4d5af5a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e946c774774d81bf04d38e4d5af5a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38585 } } }
I20260928 04:54:34.169299 9164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [sys.catalog]: This master's current role is: LEADER
I20260928 04:54:34.169179 9165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "23e946c774774d81bf04d38e4d5af5a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e946c774774d81bf04d38e4d5af5a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38585 } } }
I20260928 04:54:34.169438 9165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1 [sys.catalog]: This master's current role is: LEADER
I20260928 04:54:34.170120 9170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:54:34.170707 9170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:54:34.171271 9170 catalog_manager.cc:1259] Loaded cluster ID: 3a7a5d81a58b4e0d8ddcb6f31ea41110
I20260928 04:54:34.171319 9170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:54:34.172581 9170 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:54:34.172775 9170 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23e946c774774d81bf04d38e4d5af5a1: Loaded TSK: 0
I20260928 04:54:34.172981 9170 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 04:54:34.247536 9162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:34.247835 9162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:34.247879 9162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:34.249284 9162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:34.249397 9162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:34.249455 9162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:54:34.251214 9162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37225
--tserver_master_addrs=127.0.98.62:38585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:34.251538 9162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:34.251811 9162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:34.257671 9189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:34.257814 9186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:34.257673 9187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:34.257707 9162 server_base.cc:1034] running on GCE node
I20260928 04:54:34.258255 9162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:34.258600 9162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:34.259801 9162 hybrid_clock.cc:648] HybridClock initialized: now 1790571274259780 us; error 34 us; skew 500 ppm
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571274, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:54:34.263177 9162 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 04:54:34.263738 9162 webserver.cc:460] Webserver started at http://127.0.98.1:37225/ using document root <none> and password file <none>
I20260928 04:54:34.264014 9162 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:34.264104 9162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:34.265676 9162 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:34.266433 9196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:34.266678 9162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:34.266794 9162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "0ebe5334907f417dbaf2e23b10659a6b"
format_stamp: "Formatted at 2026-09-28 04:54:33 on dist-test-slave-m39l"
server_key: "981a6744a3daa0ac4b1431a9c8c07a7c"
server_key_iv: "0de42389ca63ac0fc5a410015b594363"
server_key_version: "encryptionkey@0"
I20260928 04:54:34.267210 9162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:34.281584 9162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:34.284420 9162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:34.284605 9162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:34.284861 9162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:54:34.285243 9162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:54:34.285302 9162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:34.285355 9162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:54:34.285382 9162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:34.293112 9162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42459
I20260928 04:54:34.293186 9309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42459 every 8 connection(s)
I20260928 04:54:34.293594 9162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:54:34.302281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 9162
I20260928 04:54:34.302688 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43079
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38585
--webserver_port=36963
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:34.304694 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:34.294442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:43199)
0928 04:54:34.294686 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:34.294750 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:34.295407 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:34.295887 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:34.295891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:34.296689 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:34.296996 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:34.297004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.298429 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.298430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:34.299911 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:34.299915 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.300617 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.300619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:34.300797 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.301508 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:34.301520 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:34.304487 (+ 2967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":125,"threads_started":1}
W20260928 04:54:34.304937 9310 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38585 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38585: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260928 04:54:34.305358 9313 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:34.294590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43199 (local address 127.0.98.62:38585)
0928 04:54:34.294786 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:34.294797 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:34.294841 (+ 44us) server_negotiation.cc:407] Connection header received
0928 04:54:34.295637 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:34.295643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:34.295690 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:34.295839 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:34.297165 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.298259 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.300035 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.300497 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.300608 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.304677 (+ 4069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43199: BlockingRecv error: recv got EOF from 127.0.98.1:43199 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":84,"threads_started":1}
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:34.309978 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:34.305202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:40253)
0928 04:54:34.305405 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:34.305418 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:34.305467 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:34.305621 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:34.305622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:34.305762 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:34.305811 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:34.305813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.306486 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.306546 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:34.307147 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:34.307149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.307546 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.307547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:34.307620 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.308098 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:34.308103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:34.309876 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171}
W20260928 04:54:34.310027 9313 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:34.305258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40253 (local address 127.0.98.62:38585)
0928 04:54:34.305508 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:34.305510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:34.305519 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:54:34.305544 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:34.305545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:34.305559 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:34.305607 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:34.305906 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.306397 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.307263 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.307460 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.307538 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.309960 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40253: BlockingRecv error: recv got EOF from 127.0.98.1:40253 (error 108)
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":83,"threads_started":1}
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:34.314796 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:34.310290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:41883)
0928 04:54:34.310492 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:34.310531 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:34.310584 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:34.310730 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:34.310731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:34.310867 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:34.310903 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:34.310904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.311548 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.311549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:34.312089 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:34.312090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.312444 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.312445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:34.312495 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.312948 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:34.312953 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:34.314705 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187}
W20260928 04:54:34.314841 9313 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:34.310436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41883 (local address 127.0.98.62:38585)
0928 04:54:34.310517 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:34.310519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:34.310584 (+ 65us) server_negotiation.cc:407] Connection header received
0928 04:54:34.310645 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:34.310646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:34.310656 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:34.310728 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:34.310976 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.311442 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.312174 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.312361 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.312445 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.314780 (+ 2335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41883: BlockingRecv error: recv got EOF from 127.0.98.1:41883 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 04:54:34.314980 9310 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:54:34.391703 9314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:34.391945 9314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:34.391974 9314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:34.393323 9314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:34.393380 9314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:34.393410 9314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:54:34.394969 9314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36963
--tserver_master_addrs=127.0.98.62:38585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:34.395244 9314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:34.395517 9314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:34.402199 9321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:34.402165 9320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:34.402210 9314 server_base.cc:1034] running on GCE node
W20260928 04:54:34.402165 9323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:34.402812 9314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:34.403156 9314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:34.404366 9314 hybrid_clock.cc:648] HybridClock initialized: now 1790571274404336 us; error 54 us; skew 500 ppm
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571274, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:54:34.407510 9314 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 04:54:34.408022 9314 webserver.cc:460] Webserver started at http://127.0.98.2:36963/ using document root <none> and password file <none>
I20260928 04:54:34.408244 9314 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:34.408308 9314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:34.410015 9314 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:54:34.410861 9330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:34.411194 9314 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:34.411279 9314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "0e83419bc72749ecb97c941f03e4efc4"
format_stamp: "Formatted at 2026-09-28 04:54:33 on dist-test-slave-m39l"
server_key: "f21041281c760e22b57730d5fd59c873"
server_key_iv: "6259a549153bbe436e9f1c6d08e82ecc"
server_key_version: "encryptionkey@0"
I20260928 04:54:34.411722 9314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:34.429790 9314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:34.432296 9314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:34.432482 9314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:34.432806 9314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:54:34.433218 9314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:54:34.433257 9314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:34.433285 9314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:54:34.433339 9314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:34.440266 9314 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43079
I20260928 04:54:34.440364 9443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43079 every 8 connection(s)
I20260928 04:54:34.440709 9314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:34.449313 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:34.441610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:50081)
0928 04:54:34.441867 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:34.441915 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:34.443078 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:34.443338 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:34.443342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:34.443625 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:34.443907 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:34.443912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.444672 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.444673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:34.445644 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:34.445646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.446042 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.446043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:34.446170 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.446689 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:34.446697 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:34.449131 (+ 2434us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":110,"threads_started":1}
W20260928 04:54:34.449331 9313 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:34.441604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50081 (local address 127.0.98.62:38585)
0928 04:54:34.441678 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:34.441680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:34.442013 (+ 333us) server_negotiation.cc:407] Connection header received
0928 04:54:34.443196 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:34.443198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:34.443213 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:34.443271 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:34.444004 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.444546 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.445741 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.445935 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.446000 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.449248 (+ 3248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50081: BlockingRecv error: recv got EOF from 127.0.98.2:50081 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 04:54:34.449440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 9314
I20260928 04:54:34.449685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45611
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38585
--webserver_port=38835
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:54:34.450121 9444 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38585 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38585: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:34.455470 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:34.450539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:34533)
0928 04:54:34.450614 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:34.450626 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:34.450672 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:34.450841 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:34.450843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:34.450965 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:34.451019 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:34.451020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.451720 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.451721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:34.452245 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:34.452247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.452624 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.452625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:34.452696 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.453209 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:34.453214 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:34.455337 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 04:54:34.455492 9313 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:34.450596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34533 (local address 127.0.98.62:38585)
0928 04:54:34.450660 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:34.450663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:34.450672 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:54:34.450722 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:34.450723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:34.450734 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:34.450797 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:34.451123 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.451630 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.452350 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.452538 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.452623 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.455436 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34533: BlockingRecv error: recv got EOF from 127.0.98.2:34533 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:34.461355 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:34.455791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:41533)
0928 04:54:34.455838 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:34.455848 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:34.455888 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:34.456051 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:34.456052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:34.456164 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:34.456200 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:34.456202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.457208 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.457209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:34.457831 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:34.457834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.458474 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.458475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:34.458595 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.459166 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:34.459171 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:34.461272 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 04:54:34.461483 9313 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:34.455838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41533 (local address 127.0.98.62:38585)
0928 04:54:34.455916 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:34.455918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:34.455929 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:54:34.455955 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:34.455956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:34.455968 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:34.456041 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:34.456276 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.457093 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.457940 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.458279 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.458364 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.461385 (+ 3021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41533: BlockingRecv error: recv got EOF from 127.0.98.2:41533 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 04:54:34.461505 9444 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:54:34.539073 9447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:34.539327 9447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:34.539356 9447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:34.540756 9447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:34.540820 9447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:34.540851 9447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:54:34.542579 9447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38835
--tserver_master_addrs=127.0.98.62:38585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:34.542876 9447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:34.543138 9447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:34.548772 9453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:34.548751 9452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:34.548940 9447 server_base.cc:1034] running on GCE node
W20260928 04:54:34.548758 9455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:34.549293 9447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:34.549651 9447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:34.550825 9447 hybrid_clock.cc:648] HybridClock initialized: now 1790571274550811 us; error 28 us; skew 500 ppm
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571274, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:54:34.553658 9447 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 04:54:34.554204 9447 webserver.cc:460] Webserver started at http://127.0.98.3:38835/ using document root <none> and password file <none>
I20260928 04:54:34.554479 9447 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:34.554630 9447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:34.556233 9447 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:54:34.556927 9462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:34.557129 9447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:54:34.557240 9447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "da17231974834159b0421cc278683811"
format_stamp: "Formatted at 2026-09-28 04:54:33 on dist-test-slave-m39l"
server_key: "ea24038770eb7d45efb79f8b3d1ec035"
server_key_iv: "b3225c8a9b7070cfdb2d7adb6aec608f"
server_key_version: "encryptionkey@0"
I20260928 04:54:34.557608 9447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:34.574373 9447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:34.576717 9447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:34.576946 9447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:34.577209 9447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:54:34.577541 9447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:54:34.577600 9447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:34.577643 9447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:54:34.577687 9447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:34.584496 9447 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45611
I20260928 04:54:34.584532 9575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45611 every 8 connection(s)
I20260928 04:54:34.584970 9447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:54:34.585675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 9447
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 04:54:34.593056 9313 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:34.585781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48573 (local address 127.0.98.62:38585)
0928 04:54:34.585838 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:34.585841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:34.586188 (+ 347us) server_negotiation.cc:407] Connection header received
0928 04:54:34.586834 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:34.586835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:34.586847 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:34.586902 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:34.587873 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.588353 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.589567 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.589775 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.589864 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.592998 (+ 3134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48573: BlockingRecv error: recv got EOF from 127.0.98.3:48573 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 04:54:34.593289 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:34.585803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.3:48573)
0928 04:54:34.586032 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:34.586088 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:34.586728 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:34.586979 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:34.586983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:34.587576 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:34.587773 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:34.587778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.588460 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.588461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:34.589463 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:34.589465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.589878 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.589879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:34.590009 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.590559 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:34.590570 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:34.592915 (+ 2345us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":108,"threads_started":1}
I20260928 04:54:34.593497 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:34.586225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.0.98.62:38585)
0928 04:54:34.586292 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:34.586294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:34.586305 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:54:34.586411 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:34.586412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:34.586425 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:34.586553 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:34.587178 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.587713 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.588512 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.588732 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.588822 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.590175 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:34.590180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:34.590180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:34.590194 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:34.591950 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:34.592372 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:34.592376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:34.592377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:34.592482 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:34.592701 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:34.592702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:34.592702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:34.592953 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:34.593142 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:34.593260 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 04:54:34.593378 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 04:54:34.593591 9576 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38585 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38585: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:34.598541 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:34.593803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.3:60705)
0928 04:54:34.593890 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:34.593901 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:34.593950 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:34.594174 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:34.594175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:34.594284 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:34.594331 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:34.594333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.595113 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.595114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:34.595617 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:34.595619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.595967 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.595967 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:34.596016 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.596517 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:34.596522 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:34.598422 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 04:54:34.598598 9315 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:34.593824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60705 (local address 127.0.98.62:38585)
0928 04:54:34.593858 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:34.593860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:34.593953 (+ 93us) server_negotiation.cc:407] Connection header received
0928 04:54:34.594021 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:34.594021 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:34.594033 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:34.594101 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:34.594430 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.595041 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.595701 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.595892 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.595949 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.598535 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60705: BlockingRecv error: recv got EOF from 127.0.98.3:60705 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:34 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:34.603173 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:34.598936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.3:38127)
0928 04:54:34.598993 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:34.599007 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:34.599077 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:34.599205 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:34.599206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:34.599333 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:34.599367 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:34.599368 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.600002 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.600003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:34.600514 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:34.600516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.600908 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.600909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:34.600963 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.601390 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:34.601395 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:34.603084 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 04:54:34.603214 9315 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:34.599008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38127 (local address 127.0.98.62:38585)
0928 04:54:34.599062 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:34.599064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:34.599074 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:54:34.599141 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:34.599141 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:34.599150 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:34.599198 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:34.599443 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.599922 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.600599 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:34.600795 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:34.600877 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:34.603154 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38127: BlockingRecv error: recv got EOF from 127.0.98.3:38127 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 04:54:34.603416 9576 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 04:54:35 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:35 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:35.320994 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:35.315607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:37133)
0928 04:54:35.315750 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:35.315764 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:35.315883 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:35.316072 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:35.316073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:35.316275 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:35.316335 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:35.316337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:35.317165 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:35.317167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:35.317785 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:35.317788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:35.318266 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:35.318267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:35.318345 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:35.318879 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:35.318884 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:35.320879 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
W20260928 04:54:35.321069 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:35.315657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37133 (local address 127.0.98.62:38585)
0928 04:54:35.315806 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:35.315809 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:35.315861 (+ 52us) server_negotiation.cc:407] Connection header received
0928 04:54:35.315967 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:35.315968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:35.315984 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:35.316063 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:35.316448 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:35.317047 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:35.317903 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:35.318156 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:35.318243 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:35.320987 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37133: BlockingRecv error: recv got EOF from 127.0.98.1:37133 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
Sep 28 04:54:35 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:35 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:35.467219 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:35.461935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:43591)
0928 04:54:35.462104 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:35.462115 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:35.462179 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:35.462383 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:35.462384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:35.462669 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:35.462720 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:35.462722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:35.463478 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:35.463479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:35.464025 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:35.464027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:35.464408 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:35.464410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:35.464473 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:35.464979 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:35.464984 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:35.467104 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
W20260928 04:54:35.467269 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:35.462035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43591 (local address 127.0.98.62:38585)
0928 04:54:35.462106 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:35.462108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:35.462191 (+ 83us) server_negotiation.cc:407] Connection header received
0928 04:54:35.462268 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:35.462269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:35.462281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:35.462372 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:35.462808 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:35.463327 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:35.464122 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:35.464319 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:35.464410 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:35.467199 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43591: BlockingRecv error: recv got EOF from 127.0.98.2:43591 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 04:54:35 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:35 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:35.609452 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:35.603922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.3:52775)
0928 04:54:35.604116 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:35.604130 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:35.604235 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:35.604445 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:35.604446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:35.604625 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:35.604675 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:35.604677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:35.605704 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:35.605705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:35.606254 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:35.606256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:35.606729 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:35.606730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:35.606792 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:35.607284 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:35.607289 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:35.609348 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":107,"threads_started":1}
W20260928 04:54:35.609485 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:35.604045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52775 (local address 127.0.98.62:38585)
0928 04:54:35.604120 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:35.604123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:35.604214 (+ 91us) server_negotiation.cc:407] Connection header received
0928 04:54:35.604324 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:35.604325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:35.604340 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:35.604396 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:35.604788 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:35.605628 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:35.606346 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:35.606641 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:35.606725 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:35.609407 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52775: BlockingRecv error: recv got EOF from 127.0.98.3:52775 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 04:54:36 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:36 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:36.327054 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:36.321710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:47531)
0928 04:54:36.321900 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:36.321914 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:36.322000 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:36.322230 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:36.322231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:36.322560 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:36.322628 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:36.322630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:36.323398 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:36.323399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:36.324003 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:36.324005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:36.324402 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:36.324404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:36.324454 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:36.324971 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:36.324976 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:36.326989 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
W20260928 04:54:36.327253 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:36.321819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47531 (local address 127.0.98.62:38585)
0928 04:54:36.321955 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:36.321957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:36.321968 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:54:36.322099 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:36.322100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:36.322114 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:36.322194 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:36.322702 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:36.323261 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:36.324084 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:36.324277 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:36.324329 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:36.327166 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47531: BlockingRecv error: recv got EOF from 127.0.98.1:47531 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
Sep 28 04:54:36 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:36 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:36.472980 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:36.467842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:37299)
0928 04:54:36.468023 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:36.468043 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:36.468100 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:36.468224 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:36.468225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:36.468404 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:36.468488 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:36.468497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:36.469278 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:36.469279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:36.469922 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:36.469925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:36.470373 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:36.470375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:36.470444 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:36.470985 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:36.470991 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:36.472918 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":93,"threads_started":1}
W20260928 04:54:36.473150 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:36.467875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37299 (local address 127.0.98.62:38585)
0928 04:54:36.467945 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:36.467948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:36.468098 (+ 150us) server_negotiation.cc:407] Connection header received
0928 04:54:36.468145 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:36.468145 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:36.468156 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:36.468207 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:36.468576 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:36.469170 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:36.470056 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:36.470277 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:36.470341 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:36.473043 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37299: BlockingRecv error: recv got EOF from 127.0.98.2:37299 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 04:54:36 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:36 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:36.615679 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:36.610076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.3:56361)
0928 04:54:36.610248 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:36.610261 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:36.610324 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:36.610633 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:36.610634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:36.610843 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:36.610897 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:36.610899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:36.611710 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:36.611711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:36.612545 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:36.612548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:36.612925 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:36.612926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:36.612989 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:36.613500 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:36.613504 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:36.615570 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":123,"threads_started":1}
W20260928 04:54:36.615731 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:36.610293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56361 (local address 127.0.98.62:38585)
0928 04:54:36.610435 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:36.610437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:36.610445 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:54:36.610468 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:36.610469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:36.610481 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:36.610590 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:36.610995 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:36.611596 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:36.612620 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:36.612836 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:36.612898 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:36.615656 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56361: BlockingRecv error: recv got EOF from 127.0.98.3:56361 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 28 04:54:37 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:37 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:37.332644 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:37.327714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:43955)
0928 04:54:37.327838 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:37.327852 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:37.327914 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:37.328120 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:37.328121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:37.328311 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:37.328444 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:37.328447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:37.329173 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:37.329174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:37.329711 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:37.329713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:37.330102 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:37.330103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:37.330151 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:37.330722 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:37.330727 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:37.332577 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
W20260928 04:54:37.332798 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:37.327795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43955 (local address 127.0.98.62:38585)
0928 04:54:37.327954 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:37.327956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:37.327964 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:54:37.327993 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:37.327994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:37.328003 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:37.328095 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:37.328548 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:37.329080 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:37.329813 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:37.330023 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:37.330087 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:37.332695 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43955: BlockingRecv error: recv got EOF from 127.0.98.1:43955 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
Sep 28 04:54:37 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:37 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:37.478794 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:37.473538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:58833)
0928 04:54:37.473733 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:37.473747 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:37.473849 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:37.474005 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:37.474006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:37.474195 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:37.474244 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:37.474245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:37.475130 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:37.475132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:37.475682 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:37.475684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:37.476059 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:37.476060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:37.476116 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:37.476598 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:37.476603 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:37.478688 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":56,"threads_started":1}
W20260928 04:54:37.478888 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:37.473760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58833 (local address 127.0.98.62:38585)
0928 04:54:37.473834 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:37.473837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:37.473847 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:54:37.473913 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:37.473914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:37.473928 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:37.474000 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:37.474400 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:37.475044 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:37.475778 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:37.475983 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:37.476062 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:37.478818 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58833: BlockingRecv error: recv got EOF from 127.0.98.2:58833 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 04:54:37 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:37 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:37.621402 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:37.616204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.3:59463)
0928 04:54:37.616409 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:37.616419 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:37.616483 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:37.616635 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:37.616636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:37.616795 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:37.616854 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:37.616855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:37.617561 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:37.617562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:37.618184 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:37.618187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:37.618658 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:37.618660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:37.618734 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:37.619286 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:37.619291 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:37.621334 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":107,"threads_started":1}
W20260928 04:54:37.621591 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:37.616307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59463 (local address 127.0.98.62:38585)
0928 04:54:37.616377 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:37.616379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:37.616487 (+ 108us) server_negotiation.cc:407] Connection header received
0928 04:54:37.616530 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:37.616531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:37.616543 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:37.616608 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:37.616921 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:37.617458 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:37.618304 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:37.618557 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:37.618644 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:37.621518 (+ 2874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59463: BlockingRecv error: recv got EOF from 127.0.98.3:59463 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 04:54:38 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:38 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:38.347121 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:38.333325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:42733)
0928 04:54:38.333564 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:38.333577 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:38.333658 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:38.333850 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:38.333851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:38.334060 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:38.334108 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:38.334109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:38.335061 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:38.335062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:38.335868 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:38.335870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:38.336296 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:38.336297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:38.336345 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:38.344597 (+ 8252us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:38.344604 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:38.347058 (+ 2454us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":163,"threads_started":1}
W20260928 04:54:38.347308 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:38.333462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42733 (local address 127.0.98.62:38585)
0928 04:54:38.333600 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:38.333602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:38.333641 (+ 39us) server_negotiation.cc:407] Connection header received
0928 04:54:38.333707 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:38.333707 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:38.333718 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:38.333783 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:38.334267 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:38.334984 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:38.335961 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:38.336155 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:38.336221 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:38.347202 (+ 10981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42733: BlockingRecv error: recv got EOF from 127.0.98.1:42733 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
Sep 28 04:54:38 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:38 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:38.484894 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:38.479397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:33573)
0928 04:54:38.479620 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:38.479631 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:38.479700 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:38.479864 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:38.479865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:38.480054 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:38.480105 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:38.480107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:38.480886 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:38.480887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:38.481492 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:38.481495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:38.481986 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:38.481987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:38.482051 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:38.482687 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:38.482692 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:38.484833 (+ 2141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":126,"threads_started":1}
W20260928 04:54:38.485126 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:38.479573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33573 (local address 127.0.98.62:38585)
0928 04:54:38.479642 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:38.479644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:38.479714 (+ 70us) server_negotiation.cc:407] Connection header received
0928 04:54:38.479759 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:38.479760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:38.479772 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:38.479837 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:38.480177 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:38.480722 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:38.481574 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:38.481895 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:38.481958 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:38.485068 (+ 3110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33573: BlockingRecv error: recv got EOF from 127.0.98.2:33573 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 04:54:38 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:38 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:38.627219 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:38.621970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.3:51277)
0928 04:54:38.622213 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:38.622229 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:38.622334 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:38.622606 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:38.622607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:38.622795 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:38.622850 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:38.622852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:38.623539 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:38.623540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:38.624110 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:38.624112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:38.624496 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:38.624498 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:38.624564 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:38.625060 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:38.625064 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:38.627109 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":128,"threads_started":1}
W20260928 04:54:38.627280 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:38.622143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51277 (local address 127.0.98.62:38585)
0928 04:54:38.622229 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:38.622232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:38.622312 (+ 80us) server_negotiation.cc:407] Connection header received
0928 04:54:38.622434 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:38.622435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:38.622449 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:38.622590 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:38.622940 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:38.623447 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:38.624205 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:38.624407 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:38.624496 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:38.627209 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51277: BlockingRecv error: recv got EOF from 127.0.98.3:51277 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 04:54:39 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:39 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:39.353029 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:39.347648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:34139)
0928 04:54:39.347785 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:39.347798 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:39.347864 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:39.348043 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:39.348044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:39.348243 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:39.348350 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:39.348352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:39.349146 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:39.349147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:39.349693 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:39.349695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:39.350080 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:39.350081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:39.350125 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:39.350752 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:39.350762 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:39.352933 (+ 2171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":104,"threads_started":1}
W20260928 04:54:39.353114 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:39.347708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34139 (local address 127.0.98.62:38585)
0928 04:54:39.347843 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:39.347846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:39.347856 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:54:39.347922 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:39.347923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:39.347933 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:39.348003 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:39.348475 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:39.349047 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:39.349794 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:39.349999 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:39.350065 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:39.353039 (+ 2974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34139: BlockingRecv error: recv got EOF from 127.0.98.1:34139 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
Sep 28 04:54:39 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:39 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:39.490798 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:39.485503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:37281)
0928 04:54:39.485660 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:39.485674 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:39.485730 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:39.485967 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:39.485968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:39.486168 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:39.486219 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:39.486220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:39.487131 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:39.487133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:39.487757 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:39.487761 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:39.488172 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:39.488173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:39.488225 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:39.488787 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:39.488792 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:39.490724 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
W20260928 04:54:39.490989 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:39.485715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37281 (local address 127.0.98.62:38585)
0928 04:54:39.485791 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:39.485793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:39.485803 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:54:39.485832 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:39.485833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:39.485845 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:39.485914 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:39.486477 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:39.487022 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:39.487867 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:39.488065 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:39.488131 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:39.490921 (+ 2790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37281: BlockingRecv error: recv got EOF from 127.0.98.2:37281 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 04:54:39 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:39 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:39.633052 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:39.628048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.3:60945)
0928 04:54:39.628302 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:39.628316 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:39.628407 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:39.628590 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:39.628591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:39.628778 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:39.628835 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:39.628837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:39.629542 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:39.629543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:39.630110 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:39.630112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:39.630537 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:39.630538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:39.630600 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:39.631078 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:39.631082 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:39.632952 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":110,"threads_started":1}
W20260928 04:54:39.633116 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:39.628184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60945 (local address 127.0.98.62:38585)
0928 04:54:39.628346 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:39.628349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:39.628384 (+ 35us) server_negotiation.cc:407] Connection header received
0928 04:54:39.628479 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:39.628480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:39.628502 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:39.628570 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:39.628919 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:39.629449 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:39.630213 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:39.630413 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:39.630490 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:39.633039 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60945: BlockingRecv error: recv got EOF from 127.0.98.3:60945 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 04:54:40 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:40 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:40.359098 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:40.353629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:41371)
0928 04:54:40.353793 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:40.353811 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:40.353865 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:40.354056 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:40.354057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:40.354221 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:40.354268 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:40.354270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:40.355148 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:40.355150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:40.355717 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:40.355719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:40.356199 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:40.356201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:40.356271 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:40.356851 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:40.356890 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:40.358988 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":106,"threads_started":1}
W20260928 04:54:40.359154 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:40.353772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41371 (local address 127.0.98.62:38585)
0928 04:54:40.353899 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:40.353905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:40.353914 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:54:40.353940 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:40.353941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:40.353952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:40.354036 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:40.354450 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:40.355044 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:40.355851 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:40.356093 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:40.356157 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:40.359079 (+ 2922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41371: BlockingRecv error: recv got EOF from 127.0.98.1:41371 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
Sep 28 04:54:40 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:40 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:40.496150 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:40.491353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:56481)
0928 04:54:40.491492 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:40.491511 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:40.491566 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:40.491697 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:40.491697 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:40.491882 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:40.491931 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:40.491933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:40.492658 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:40.492659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:40.493198 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:40.493200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:40.493560 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:40.493562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:40.493615 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:40.494087 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:40.494092 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:40.496088 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
W20260928 04:54:40.496307 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:40.491431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56481 (local address 127.0.98.62:38585)
0928 04:54:40.491497 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:40.491499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:40.491561 (+ 62us) server_negotiation.cc:407] Connection header received
0928 04:54:40.491617 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:40.491618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:40.491627 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:40.491670 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:40.492041 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:40.492571 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:40.493286 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:40.493480 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:40.493537 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:40.496213 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56481: BlockingRecv error: recv got EOF from 127.0.98.2:56481 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 04:54:40 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:40 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:40.639273 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:40.633748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.3:50767)
0928 04:54:40.633929 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:40.633945 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:40.634045 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:40.634230 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:40.634231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:40.634468 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:40.634556 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:40.634558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:40.635390 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:40.635391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:40.635950 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:40.635952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:40.636404 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:40.636406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:40.636481 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:40.637008 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:40.637012 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:40.639154 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":106,"threads_started":1}
W20260928 04:54:40.639393 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:40.633812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50767 (local address 127.0.98.62:38585)
0928 04:54:40.633908 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:40.633911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:40.634026 (+ 115us) server_negotiation.cc:407] Connection header received
0928 04:54:40.634135 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:40.634136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:40.634150 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:40.634223 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:40.634674 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:40.635304 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:40.636049 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:40.636259 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:40.636349 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:40.639316 (+ 2967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50767: BlockingRecv error: recv got EOF from 127.0.98.3:50767 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 04:54:41 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:41 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:41.365621 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:41.359734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:36305)
0928 04:54:41.359892 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:41.359905 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:41.359967 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:41.360468 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:41.360469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:41.360710 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:41.360771 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:41.360774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:41.361561 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:41.361562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:41.362108 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:41.362110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:41.362513 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:41.362514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:41.362569 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:41.363145 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:41.363151 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:41.365504 (+ 2353us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
W20260928 04:54:41.365658 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:41.359836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36305 (local address 127.0.98.62:38585)
0928 04:54:41.359980 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:41.359982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:41.359996 (+ 14us) server_negotiation.cc:407] Connection header received
0928 04:54:41.360180 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:41.360181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:41.360191 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:41.360450 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:41.360865 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:41.361477 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:41.362197 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:41.362405 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:41.362542 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:41.365589 (+ 3047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36305: BlockingRecv error: recv got EOF from 127.0.98.1:36305 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
Sep 28 04:54:41 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:41 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:41.501646 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:41.496809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:33375)
0928 04:54:41.496985 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:41.497003 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:41.497062 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:41.497229 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:41.497230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:41.497428 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:41.497480 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:41.497482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:41.498202 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:41.498203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:41.498849 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:41.498851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:41.499198 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:41.499198 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:41.499285 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:41.499765 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:41.499770 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:41.501585 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":119,"threads_started":1}
W20260928 04:54:41.501768 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:41.496935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33375 (local address 127.0.98.62:38585)
0928 04:54:41.496986 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:41.496988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:41.497049 (+ 61us) server_negotiation.cc:407] Connection header received
0928 04:54:41.497128 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:41.497129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:41.497141 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:41.497188 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:41.497575 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:41.498096 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:41.498929 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:41.499125 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:41.499184 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:41.501687 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33375: BlockingRecv error: recv got EOF from 127.0.98.2:33375 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 04:54:41 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:41 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:41.644762 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:41.639887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.3:52079)
0928 04:54:41.640051 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:41.640071 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:41.640126 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:41.640309 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:41.640310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:41.640485 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:41.640538 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:41.640540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:41.641242 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:41.641243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:41.641799 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:41.641802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:41.642175 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:41.642176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:41.642228 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:41.642724 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:41.642728 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:41.644641 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":112,"threads_started":1}
W20260928 04:54:41.644801 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:41.640013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52079 (local address 127.0.98.62:38585)
0928 04:54:41.640068 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:41.640070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:41.640118 (+ 48us) server_negotiation.cc:407] Connection header received
0928 04:54:41.640173 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:41.640174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:41.640188 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:41.640262 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:41.640639 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:41.641141 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:41.641892 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:41.642081 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:41.642138 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:41.644729 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52079: BlockingRecv error: recv got EOF from 127.0.98.3:52079 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 04:54:42 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:42 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:42.372509 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:42.366280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:50279)
0928 04:54:42.366537 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:42.366549 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:42.366622 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:42.366983 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:42.366984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:42.367171 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:42.367218 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:42.367220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:42.368108 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:42.368109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:42.368918 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:42.368920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:42.369287 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:42.369289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:42.369481 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:42.370088 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:42.370094 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:42.372407 (+ 2313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":97,"threads_started":1}
W20260928 04:54:42.372557 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:42.366454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50279 (local address 127.0.98.62:38585)
0928 04:54:42.366631 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:42.366633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:42.366645 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:54:42.366840 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:42.366842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:42.366861 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:42.366940 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:42.367417 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:42.368013 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:42.368998 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:42.369206 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:42.369265 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:42.372487 (+ 3222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50279: BlockingRecv error: recv got EOF from 127.0.98.1:50279 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":102,"threads_started":1}
Sep 28 04:54:42 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:42 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:42.507227 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:42.502278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:39153)
0928 04:54:42.502446 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:42.502460 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:42.502549 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:42.502688 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:42.502689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:42.502849 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:42.502905 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:42.502907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:42.503710 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:42.503710 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:42.504248 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:42.504251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:42.504611 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:42.504612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:42.504650 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:42.505171 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:42.505176 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:42.507128 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
W20260928 04:54:42.507282 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:42.502428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39153 (local address 127.0.98.62:38585)
0928 04:54:42.502479 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:42.502481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:42.502556 (+ 75us) server_negotiation.cc:407] Connection header received
0928 04:54:42.502609 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:42.502610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:42.502621 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:42.502662 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:42.502973 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:42.503639 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:42.504325 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:42.504543 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:42.504596 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:42.507217 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39153: BlockingRecv error: recv got EOF from 127.0.98.2:39153 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 04:54:42 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:42 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:42.650027 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:42.645396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.3:33279)
0928 04:54:42.645540 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:42.645557 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:42.645615 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:42.645744 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:42.645745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:42.645957 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:42.646007 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:42.646009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:42.646648 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:42.646649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:42.647182 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:42.647183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:42.647557 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:42.647558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:42.647596 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:42.648142 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:42.648147 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:42.649969 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
W20260928 04:54:42.650158 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:42.645469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33279 (local address 127.0.98.62:38585)
0928 04:54:42.645551 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:42.645553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:42.645617 (+ 64us) server_negotiation.cc:407] Connection header received
0928 04:54:42.645673 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:42.645674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:42.645686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:42.645728 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:42.646072 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:42.646581 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:42.647285 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:42.647485 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:42.647548 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:42.650065 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33279: BlockingRecv error: recv got EOF from 127.0.98.3:33279 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 04:54:43 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:43 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:43.386699 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:43.373170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:56025)
0928 04:54:43.373348 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:43.373366 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:43.373430 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:43.373588 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:43.373589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:43.373786 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:43.373840 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:43.373842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:43.374893 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:43.374895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:43.375502 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:43.375503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:43.375849 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:43.375850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:43.375895 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:43.384204 (+ 8309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:43.384211 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:43.386604 (+ 2393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":110,"threads_started":1}
W20260928 04:54:43.386770 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:43.373233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56025 (local address 127.0.98.62:38585)
0928 04:54:43.373383 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:43.373385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:43.373417 (+ 32us) server_negotiation.cc:407] Connection header received
0928 04:54:43.373481 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:43.373482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:43.373492 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:43.373567 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:43.373929 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:43.374803 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:43.375582 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:43.375770 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:43.375831 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:43.386698 (+ 10867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56025: BlockingRecv error: recv got EOF from 127.0.98.1:56025 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
Sep 28 04:54:43 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:43 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:43.513383 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:43.507886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:34537)
0928 04:54:43.508146 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:43.508160 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:43.508284 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:43.508591 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:43.508592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:43.508766 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:43.508847 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:43.508849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:43.509648 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:43.509649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:43.510185 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:43.510187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:43.510676 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:43.510677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:43.510730 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:43.511275 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:43.511280 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:43.513280 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":162,"threads_started":1}
W20260928 04:54:43.513449 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:43.508071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34537 (local address 127.0.98.62:38585)
0928 04:54:43.508176 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:43.508178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:43.508262 (+ 84us) server_negotiation.cc:407] Connection header received
0928 04:54:43.508432 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:43.508433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:43.508445 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:43.508534 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:43.508988 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:43.509564 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:43.510291 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:43.510532 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:43.510599 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:43.513376 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34537: BlockingRecv error: recv got EOF from 127.0.98.2:34537 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 04:54:43 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:43 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:43.655925 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:43.650677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.3:50729)
0928 04:54:43.650877 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:43.650889 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:43.650965 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:43.651150 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:43.651151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:43.651471 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:43.651527 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:43.651529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:43.652228 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:43.652229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:43.652751 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:43.652753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:43.653369 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:43.653370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:43.653432 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:43.653996 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:43.654001 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:43.655866 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
W20260928 04:54:43.656061 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:43.650786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50729 (local address 127.0.98.62:38585)
0928 04:54:43.650859 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:43.650863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:43.650965 (+ 102us) server_negotiation.cc:407] Connection header received
0928 04:54:43.651024 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:43.651024 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:43.651036 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:43.651099 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:43.651608 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:43.652127 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:43.652859 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:43.653191 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:43.653266 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:43.655970 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50729: BlockingRecv error: recv got EOF from 127.0.98.3:50729 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 04:54:44 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:44 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:44.392309 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:44.387247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:53997)
0928 04:54:44.387410 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:44.387420 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:44.387472 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:44.387627 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:44.387627 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:44.387807 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:44.387851 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:44.387853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:44.388537 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:44.388538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:44.389460 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:44.389464 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:44.389857 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:44.389858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:44.389906 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:44.390364 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:44.390369 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:44.392234 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":67,"threads_started":1}
W20260928 04:54:44.392473 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:44.387294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53997 (local address 127.0.98.62:38585)
0928 04:54:44.387477 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:44.387479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:44.387517 (+ 38us) server_negotiation.cc:407] Connection header received
0928 04:54:44.387537 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:44.387537 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:44.387547 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:44.387596 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:44.387921 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:44.388444 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:44.389565 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:44.389762 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:44.389844 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:44.392365 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53997: BlockingRecv error: recv got EOF from 127.0.98.1:53997 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":130,"threads_started":1}
Sep 28 04:54:44 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:44 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:44.519393 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:44.514024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:44575)
0928 04:54:44.514205 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:44.514220 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:44.514310 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:44.514560 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:44.514561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:44.514796 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:44.514848 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:44.514850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:44.515691 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:44.515692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:44.516240 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:44.516242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:44.516646 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:44.516647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:44.516703 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:44.517219 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:44.517225 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:44.519283 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":101,"threads_started":1}
W20260928 04:54:44.519446 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:44.514175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44575 (local address 127.0.98.62:38585)
0928 04:54:44.514271 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:44.514273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:44.514289 (+ 16us) server_negotiation.cc:407] Connection header received
0928 04:54:44.514395 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:44.514396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:44.514442 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:44.514552 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:44.514990 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:44.515583 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:44.516344 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:44.516554 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:44.516644 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:44.519382 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44575: BlockingRecv error: recv got EOF from 127.0.98.2:44575 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 04:54:44 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:44 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:44.661701 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:44.656586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.3:52749)
0928 04:54:44.656764 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:44.656778 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:44.656858 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:44.657014 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:44.657015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:44.657182 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:44.657233 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:44.657235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:44.658012 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:44.658013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:44.658660 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:44.658662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:44.659045 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:44.659045 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:44.659084 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:44.659623 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:44.659628 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:44.661642 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":76,"threads_started":1}
W20260928 04:54:44.661841 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:44.656747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52749 (local address 127.0.98.62:38585)
0928 04:54:44.656858 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:44.656860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:44.656899 (+ 39us) server_negotiation.cc:407] Connection header received
0928 04:54:44.656925 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:44.656926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:44.656939 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:44.656987 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:44.657390 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:44.657935 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:44.658763 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:44.658976 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:44.659034 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:44.661750 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52749: BlockingRecv error: recv got EOF from 127.0.98.3:52749 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 04:54:45 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:45 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:45.398265 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:45.392958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:45557)
0928 04:54:45.393142 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:45.393174 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:45.393249 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:45.393457 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:45.393458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:45.393635 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:45.393684 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:45.393686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:45.394481 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:45.394483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:45.395094 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:45.395097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:45.395569 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:45.395570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:45.395627 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:45.396128 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:45.396133 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:45.398200 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":74,"threads_started":1}
W20260928 04:54:45.398437 9621 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:45.393021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45557 (local address 127.0.98.62:38585)
0928 04:54:45.393155 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:45.393157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:45.393234 (+ 77us) server_negotiation.cc:407] Connection header received
0928 04:54:45.393352 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:45.393352 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:45.393367 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:45.393443 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:45.393817 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:45.394376 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:45.395228 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:45.395457 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:45.395522 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:45.398343 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45557: BlockingRecv error: recv got EOF from 127.0.98.1:45557 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
Sep 28 04:54:45 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:45 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:45.525274 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:45.520037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:56987)
0928 04:54:45.520208 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:45.520221 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:45.520288 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:45.520487 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:45.520488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:45.520679 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:45.520727 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:45.520729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:45.521433 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:45.521434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:45.521963 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:45.521964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:45.522336 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:45.522337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:45.522394 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:45.522991 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:45.522997 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:45.525186 (+ 2189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":105,"threads_started":1}
W20260928 04:54:45.525519 9621 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:45.520203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56987 (local address 127.0.98.62:38585)
0928 04:54:45.520281 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:45.520283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:45.520295 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:54:45.520361 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:45.520362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:45.520378 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:45.520475 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:45.520816 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:45.521333 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:45.522056 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:45.522256 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:45.522338 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:45.525468 (+ 3130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56987: BlockingRecv error: recv got EOF from 127.0.98.2:56987 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 04:54:45 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:45 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:45.667536 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:45.662241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.3:43813)
0928 04:54:45.662408 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:45.662420 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:45.662477 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:45.662683 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:45.662684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:45.662862 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:45.662913 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:45.662915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:45.663614 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:45.663615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:45.664181 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:45.664183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:45.664641 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:45.664642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:45.664691 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:45.665308 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:45.665313 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:45.667474 (+ 2161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
W20260928 04:54:45.667678 9621 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:45.662278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43813 (local address 127.0.98.62:38585)
0928 04:54:45.662412 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:45.662414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:45.662461 (+ 47us) server_negotiation.cc:407] Connection header received
0928 04:54:45.662550 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:45.662550 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:45.662563 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:45.662613 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:45.662989 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:45.663537 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:45.664273 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:45.664544 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:45.664603 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:45.667586 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43813: BlockingRecv error: recv got EOF from 127.0.98.3:43813 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 04:54:46 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:46 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:46.404060 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:46.398908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:43059)
0928 04:54:46.399094 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:46.399106 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:46.399159 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:46.399411 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:46.399413 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:46.399589 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:46.399639 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:46.399641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:46.400345 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:46.400346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:46.400869 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:46.400871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:46.401226 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:46.401227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:46.401268 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:46.401851 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:46.401856 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:46.403977 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":105,"threads_started":1}
W20260928 04:54:46.404233 9625 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:46.398983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43059 (local address 127.0.98.62:38585)
0928 04:54:46.399190 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:46.399193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:46.399212 (+ 19us) server_negotiation.cc:407] Connection header received
0928 04:54:46.399313 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:46.399314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:46.399328 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:46.399403 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:46.399712 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:46.400211 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:46.400952 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:46.401152 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:46.401211 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:46.404118 (+ 2907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43059: BlockingRecv error: recv got EOF from 127.0.98.1:43059 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":85,"threads_started":1}
Sep 28 04:54:46 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:46 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:46.530857 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:46.525851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:44211)
0928 04:54:46.526000 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:46.526020 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:46.526085 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:46.526260 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:46.526261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:46.526437 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:46.526533 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:46.526535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:46.527235 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:46.527236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:46.527849 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:46.527851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:46.528213 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:46.528214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:46.528271 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:46.528751 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:46.528756 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:46.530792 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
W20260928 04:54:46.531087 9625 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:46.526017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44211 (local address 127.0.98.62:38585)
0928 04:54:46.526095 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:46.526096 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:46.526105 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:54:46.526144 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:46.526145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:46.526177 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:46.526236 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:46.526620 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:46.527146 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:46.527939 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:46.528123 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:46.528187 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:46.531026 (+ 2839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44211: BlockingRecv error: recv got EOF from 127.0.98.2:44211 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 04:54:46 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:46 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:46.673801 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:46.668257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.3:51151)
0928 04:54:46.668433 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:46.668445 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:46.668518 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:46.668681 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:46.668682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:46.668911 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:46.668959 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:46.668961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:46.669900 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:46.669902 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:46.670712 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:46.670736 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:46.671359 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:46.671360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:46.671430 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:46.671881 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:46.671886 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:46.673727 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":78,"threads_started":1}
W20260928 04:54:46.673982 9625 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:46.668344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51151 (local address 127.0.98.62:38585)
0928 04:54:46.668403 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:46.668406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:46.668499 (+ 93us) server_negotiation.cc:407] Connection header received
0928 04:54:46.668573 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:46.668574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:46.668589 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:46.668676 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:46.669060 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:46.669788 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:46.670838 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:46.671237 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:46.671356 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:46.673870 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51151: BlockingRecv error: recv got EOF from 127.0.98.3:51151 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 04:54:47 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:47 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:47.410375 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:47.404802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:48433)
0928 04:54:47.404989 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:47.405014 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:47.405095 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:47.405417 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:47.405418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:47.405589 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:47.405646 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:47.405647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:47.406375 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:47.406376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:47.407081 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:47.407084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:47.407501 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:47.407502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:47.407563 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:47.408161 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:47.408165 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:47.410306 (+ 2141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
W20260928 04:54:47.410563 9629 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:47.404978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48433 (local address 127.0.98.62:38585)
0928 04:54:47.405119 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:47.405121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:47.405147 (+ 26us) server_negotiation.cc:407] Connection header received
0928 04:54:47.405173 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:47.405173 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:47.405184 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:47.405237 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:47.405731 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:47.406255 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:47.407180 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:47.407407 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:47.407499 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:47.410430 (+ 2931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48433: BlockingRecv error: recv got EOF from 127.0.98.1:48433 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
Sep 28 04:54:47 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:47 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:47.536473 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:47.531408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:50191)
0928 04:54:47.531566 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:47.531578 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:47.531645 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:47.531809 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:47.531810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:47.531994 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:47.532043 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:47.532045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:47.532805 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:47.532805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:47.533429 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:47.533431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:47.533874 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:47.533875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:47.533918 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:47.534437 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:47.534442 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:47.536412 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
W20260928 04:54:47.536572 9629 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:47.531581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50191 (local address 127.0.98.62:38585)
0928 04:54:47.531675 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:47.531677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:47.531687 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:54:47.531717 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:47.531718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:47.531735 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:47.531791 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:47.532143 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:47.532726 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:47.533541 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:47.533807 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:47.533866 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:47.536501 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50191: BlockingRecv error: recv got EOF from 127.0.98.2:50191 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 04:54:47 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:47 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:47.679232 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:47.674306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.3:41801)
0928 04:54:47.674496 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:47.674533 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:47.674625 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:47.674819 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:47.674820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:47.674997 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:47.675044 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:47.675046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:47.675745 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:47.675747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:47.676295 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:47.676297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:47.676692 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:47.676693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:47.676760 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:47.677270 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:47.677275 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:47.679130 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
W20260928 04:54:47.679296 9629 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:47.674413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41801 (local address 127.0.98.62:38585)
0928 04:54:47.674520 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:47.674523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:47.674604 (+ 81us) server_negotiation.cc:407] Connection header received
0928 04:54:47.674697 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:47.674698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:47.674712 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:47.674813 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:47.675144 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:47.675660 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:47.676410 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:47.676618 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:47.676695 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:47.679217 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41801: BlockingRecv error: recv got EOF from 127.0.98.3:41801 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 04:54:48 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:48 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:48.421911 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:48.411037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:57325)
0928 04:54:48.411378 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:48.411395 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:48.411502 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:48.411761 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:48.411762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:48.412029 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:48.412094 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:48.412097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:48.413018 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:48.413019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:48.413575 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:48.413577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:48.413941 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:48.413942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:48.413984 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:48.419631 (+ 5647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:48.419638 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:48.421846 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":198,"threads_started":1}
W20260928 04:54:48.422124 9633 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:48.411248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57325 (local address 127.0.98.62:38585)
0928 04:54:48.411373 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:48.411375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:48.411501 (+ 126us) server_negotiation.cc:407] Connection header received
0928 04:54:48.411609 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:48.411610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:48.411642 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:48.411719 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:48.412188 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:48.412921 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:48.413661 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:48.413853 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:48.413917 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:48.422035 (+ 8118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57325: BlockingRecv error: recv got EOF from 127.0.98.1:57325 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
Sep 28 04:54:48 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:48 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:48.542657 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:48.537018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:50055)
0928 04:54:48.537165 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:48.537181 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:48.537245 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:48.537518 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:48.537519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:48.537702 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:48.537753 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:48.537755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:48.538754 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:48.538755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:48.539335 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:48.539338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:48.539772 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:48.539773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:48.539827 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:48.540413 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:48.540418 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:48.542589 (+ 2171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
W20260928 04:54:48.542807 9633 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:48.537127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50055 (local address 127.0.98.62:38585)
0928 04:54:48.537270 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:48.537272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:48.537282 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:54:48.537308 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:48.537308 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:48.537319 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:48.537375 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:48.537892 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:48.538655 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:48.539447 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:48.539676 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:48.539744 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:48.542719 (+ 2975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50055: BlockingRecv error: recv got EOF from 127.0.98.2:50055 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 28 04:54:48 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:48 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:48.684885 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:48.679879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.3:49645)
0928 04:54:48.680018 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:48.680032 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:48.680103 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:48.680298 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:48.680299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:48.680516 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:48.680575 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:48.680577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:48.681301 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:48.681302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:48.681863 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:48.681866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:48.682323 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:48.682324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:48.682362 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:48.682931 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:48.682936 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:48.684816 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
W20260928 04:54:48.685036 9633 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:48.679918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49645 (local address 127.0.98.62:38585)
0928 04:54:48.679976 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:48.679979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:48.680141 (+ 162us) server_negotiation.cc:407] Connection header received
0928 04:54:48.680177 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:48.680178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:48.680195 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:48.680279 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:48.680658 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:48.681193 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:48.681966 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:48.682242 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:48.682323 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:48.684916 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49645: BlockingRecv error: recv got EOF from 127.0.98.3:49645 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 04:54:49 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:49 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:49.428040 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:49.422671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.1:50351)
0928 04:54:49.422854 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:49.422868 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:49.422934 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:49.423186 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:49.423187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:49.423407 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:49.423454 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:49.423456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:49.424171 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:49.424172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:49.424742 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:49.424744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:49.425128 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:49.425129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:49.425167 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:49.425743 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:49.425748 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:49.427941 (+ 2193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":143,"threads_started":1}
W20260928 04:54:49.428211 9637 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:49.422860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50351 (local address 127.0.98.62:38585)
0928 04:54:49.422987 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:49.422989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:49.423004 (+ 15us) server_negotiation.cc:407] Connection header received
0928 04:54:49.423038 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:49.423038 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:49.423048 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:49.423133 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:49.423566 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:49.424087 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:49.424845 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:49.425058 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:49.425119 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:49.428101 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50351: BlockingRecv error: recv got EOF from 127.0.98.1:50351 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
Sep 28 04:54:49 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:54:49 dist-test-slave-m39l krb5kdc[8579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:54:49.548769 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:49.543311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38585 (local address 127.0.98.2:57351)
0928 04:54:49.543467 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:49.543477 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:49.543526 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:49.543759 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:49.543760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:49.543926 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:49.543975 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:49.543977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:49.544746 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:49.544748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:49.545298 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:49.545300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:49.545716 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:49.545717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:49.545774 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:49.546431 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:49.546436 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:49.548677 (+ 2241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38585: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
W20260928 04:54:49.548914 9637 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:49.543507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57351 (local address 127.0.98.62:38585)
0928 04:54:49.543583 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:49.543586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:49.543594 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:54:49.543622 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:49.543623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:49.543634 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:49.543695 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:49.544103 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:49.544629 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:49.545412 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:49.545614 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:49.545676 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:49.548834 (+ 3158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57351: BlockingRecv error: recv got EOF from 127.0.98.2:57351 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 04:54:49.596375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 9162
I20260928 04:54:49.601512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 9314
I20260928 04:54:49.605827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 9447
I20260928 04:54:49.610356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 9092
2026-09-28T04:54:49Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17658 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T04:54:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:54:49Z Disabled control of system clock
I20260928 04:54:49.639879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38505
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 04:54:49.724956 9642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:49.725185 9642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:49.725208 9642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:49.726672 9642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:54:49.726732 9642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:49.726744 9642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:49.726758 9642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:54:49.726770 9642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:54:49.728399 9642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38505
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:49.728732 9642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:49.729091 9642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:49.735412 9650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:49.735401 9648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:49.735401 9642 server_base.cc:1034] running on GCE node
W20260928 04:54:49.735399 9647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:49.736080 9642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:49.736459 9642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:49.737699 9642 hybrid_clock.cc:648] HybridClock initialized: now 1790571289737676 us; error 49 us; skew 500 ppm
I20260928 04:54:49.739351 9642 webserver.cc:460] Webserver started at http://127.0.98.62:37779/ using document root <none> and password file <none>
I20260928 04:54:49.739639 9642 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:49.739718 9642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:49.739879 9642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:49.741041 9642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "332bdde6eaf24ef3ac7e120961b9c4a4"
format_stamp: "Formatted at 2026-09-28 04:54:49 on dist-test-slave-m39l"
server_key: "32b915122ebacd4805b0c59862e796d3"
server_key_iv: "9c0c1dcf6d6a8e4a3f403f5aafad47c4"
server_key_version: "encryptionkey@0"
I20260928 04:54:49.741425 9642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "332bdde6eaf24ef3ac7e120961b9c4a4"
format_stamp: "Formatted at 2026-09-28 04:54:49 on dist-test-slave-m39l"
server_key: "32b915122ebacd4805b0c59862e796d3"
server_key_iv: "9c0c1dcf6d6a8e4a3f403f5aafad47c4"
server_key_version: "encryptionkey@0"
I20260928 04:54:49.743046 9642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:54:49.744062 9656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:49.744467 9642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:49.744592 9642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "332bdde6eaf24ef3ac7e120961b9c4a4"
format_stamp: "Formatted at 2026-09-28 04:54:49 on dist-test-slave-m39l"
server_key: "32b915122ebacd4805b0c59862e796d3"
server_key_iv: "9c0c1dcf6d6a8e4a3f403f5aafad47c4"
server_key_version: "encryptionkey@0"
I20260928 04:54:49.744680 9642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:49.759019 9642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:49.759390 9642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:49.759554 9642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:49.764132 9642 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38505
I20260928 04:54:49.764178 9708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38505 every 8 connection(s)
I20260928 04:54:49.764585 9642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:54:49.765389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 9642
I20260928 04:54:49.765488 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:54:49.765481 9709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:49.765687 392 external_mini_cluster.cc:1428] Setting key 18933f380490e7622f9aefb248cdbcf9
I20260928 04:54:49.769593 9709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4: Bootstrap starting.
I20260928 04:54:49.770347 9709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:49.770780 9709 log.cc:824] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4: Log is configured to *not* fsync() on all Append() calls
I20260928 04:54:49.771589 9709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4: No bootstrap required, opened a new log
I20260928 04:54:49.771927 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:49.766316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.0.98.62:38505)
0928 04:54:49.766572 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:49.766579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:49.766608 (+ 29us) server_negotiation.cc:407] Connection header received
0928 04:54:49.767125 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:49.767132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:49.767199 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:49.767382 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:49.767647 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:49.768527 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:49.769714 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:49.770169 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:49.770276 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:49.771166 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:49.771172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:49.771172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:54:49.771377 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:49.771414 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:54:49.771427 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:49.771508 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:49.771595 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 04:54:49.771681 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260928 04:54:49.773838 9709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332bdde6eaf24ef3ac7e120961b9c4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38505 } }
I20260928 04:54:49.774044 9709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:49.774091 9709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332bdde6eaf24ef3ac7e120961b9c4a4, State: Initialized, Role: FOLLOWER
I20260928 04:54:49.774212 9709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332bdde6eaf24ef3ac7e120961b9c4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38505 } }
I20260928 04:54:49.774308 9709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:54:49.774343 9709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:54:49.774394 9709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:49.775060 9709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332bdde6eaf24ef3ac7e120961b9c4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38505 } }
I20260928 04:54:49.775153 9709 leader_election.cc:302] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 332bdde6eaf24ef3ac7e120961b9c4a4; no voters:
I20260928 04:54:49.775290 9709 leader_election.cc:288] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:54:49.775378 9714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:54:49.775563 9709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:54:49.775576 9714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4 [term 1 LEADER]: Becoming Leader. State: Replica: 332bdde6eaf24ef3ac7e120961b9c4a4, State: Running, Role: LEADER
I20260928 04:54:49.775735 9714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332bdde6eaf24ef3ac7e120961b9c4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38505 } }
I20260928 04:54:49.776137 9716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 332bdde6eaf24ef3ac7e120961b9c4a4. Latest consensus state: current_term: 1 leader_uuid: "332bdde6eaf24ef3ac7e120961b9c4a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332bdde6eaf24ef3ac7e120961b9c4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38505 } } }
I20260928 04:54:49.776217 9716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4 [sys.catalog]: This master's current role is: LEADER
I20260928 04:54:49.776403 9715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "332bdde6eaf24ef3ac7e120961b9c4a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332bdde6eaf24ef3ac7e120961b9c4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38505 } } }
I20260928 04:54:49.776503 9715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4 [sys.catalog]: This master's current role is: LEADER
I20260928 04:54:49.776847 9722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:54:49.777344 9722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:54:49.778931 9722 catalog_manager.cc:1347] Generated new cluster ID: 36528771263447c38b7bcb636854a4dd
I20260928 04:54:49.778986 9722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:54:49.790238 9722 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:54:49.790869 9722 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:54:49.794309 9722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 332bdde6eaf24ef3ac7e120961b9c4a4: Generated new TSK 0
I20260928 04:54:49.794538 9722 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:54:49.803546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38505
--builtin_ntp_servers=127.0.98.20:45523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 04:54:49.889671 9733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:49.889894 9733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:49.889918 9733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:49.891247 9733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:49.891296 9733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:49.891323 9733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:54:49.892736 9733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:49.893023 9733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:49.893343 9733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:49.899901 9741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:49.899953 9739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:49.899917 9738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:49.900554 9733 server_base.cc:1034] running on GCE node
I20260928 04:54:49.900780 9733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:49.901049 9733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:49.902231 9733 hybrid_clock.cc:648] HybridClock initialized: now 1790571289902214 us; error 28 us; skew 500 ppm
I20260928 04:54:49.903788 9733 webserver.cc:460] Webserver started at http://127.0.98.1:45755/ using document root <none> and password file <none>
I20260928 04:54:49.904069 9733 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:49.904127 9733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:49.904261 9733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:49.905773 9733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "07c0472b45224351a6a8a9c9706a65f3"
format_stamp: "Formatted at 2026-09-28 04:54:49 on dist-test-slave-m39l"
server_key: "ea9ee3971516e803d25f2c3514e5ddbd"
server_key_iv: "9c418feef55d1c2aa271dbf97620e1d9"
server_key_version: "encryptionkey@0"
I20260928 04:54:49.906206 9733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "07c0472b45224351a6a8a9c9706a65f3"
format_stamp: "Formatted at 2026-09-28 04:54:49 on dist-test-slave-m39l"
server_key: "ea9ee3971516e803d25f2c3514e5ddbd"
server_key_iv: "9c418feef55d1c2aa271dbf97620e1d9"
server_key_version: "encryptionkey@0"
I20260928 04:54:49.907924 9733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 04:54:49.908869 9747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:49.909200 9733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:49.909302 9733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "07c0472b45224351a6a8a9c9706a65f3"
format_stamp: "Formatted at 2026-09-28 04:54:49 on dist-test-slave-m39l"
server_key: "ea9ee3971516e803d25f2c3514e5ddbd"
server_key_iv: "9c418feef55d1c2aa271dbf97620e1d9"
server_key_version: "encryptionkey@0"
I20260928 04:54:49.909381 9733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:49.937758 9733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:49.938117 9733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:49.938373 9733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:49.938659 9733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:54:49.939047 9733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:54:49.939085 9733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:49.939108 9733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:54:49.939121 9733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:49.945140 9733 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41029
I20260928 04:54:49.945226 9860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41029 every 8 connection(s)
I20260928 04:54:49.945585 9733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:54:49.949396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 9733
I20260928 04:54:49.949477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:54:49.949714 392 external_mini_cluster.cc:1428] Setting key c0b4c9bd3f3cc229f875061f3ecff797
I20260928 04:54:49.950861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38505
--builtin_ntp_servers=127.0.98.20:45523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 04:54:49.951905 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:49.946402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58281 (local address 127.0.98.62:38505)
0928 04:54:49.946539 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:49.946542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:49.946777 (+ 235us) server_negotiation.cc:407] Connection header received
0928 04:54:49.947356 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:49.947357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:49.947374 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:49.947440 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:49.948209 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:49.948833 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:49.950136 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:49.950365 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:49.950470 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:49.951563 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:49.951565 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:49.951565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:54:49.951571 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:49.951589 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:54:49.951593 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:49.951656 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:49.951741 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 04:54:49.951787 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 04:54:49.952040 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:49.946357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38505 (local address 127.0.98.1:58281)
0928 04:54:49.946602 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:49.946647 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:49.947241 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:49.947545 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:49.947549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:49.947560 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:49.948105 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:49.948110 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:49.948938 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:49.948939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:49.949960 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:49.949963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:49.950564 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:49.950566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:49.950737 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:49.951413 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:54:49.951414 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:49.951446 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:54:49.951471 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:49.951663 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:49.951667 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:49.951670 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:54:49.951725 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 04:54:49.951817 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
I20260928 04:54:49.952636 9861 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38505
I20260928 04:54:49.952736 9861 heartbeater.cc:464] Registering TS with master...
I20260928 04:54:49.952996 9861 heartbeater.cc:511] Master 127.0.98.62:38505 requested a full tablet report, sending...
I20260928 04:54:49.953578 9673 ts_manager.cc:194] Registered new tserver with Master: 07c0472b45224351a6a8a9c9706a65f3 (127.0.98.1:41029)
I20260928 04:54:49.954588 9673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58281
W20260928 04:54:50.040393 9864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:50.040634 9864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:50.040658 9864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:50.041867 9864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:50.041914 9864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:50.041940 9864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:54:50.043519 9864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:50.043746 9864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:50.044035 9864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:50.049588 9872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:50.049605 9870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:50.049605 9869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:50.049672 9864 server_base.cc:1034] running on GCE node
I20260928 04:54:50.050032 9864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:50.050252 9864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:50.051422 9864 hybrid_clock.cc:648] HybridClock initialized: now 1790571290051377 us; error 27 us; skew 500 ppm
I20260928 04:54:50.052814 9864 webserver.cc:460] Webserver started at http://127.0.98.2:33563/ using document root <none> and password file <none>
I20260928 04:54:50.053104 9864 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:50.053157 9864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:50.053323 9864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:50.054633 9864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "97c8a0ccec8b428597ae8467849e83d0"
format_stamp: "Formatted at 2026-09-28 04:54:50 on dist-test-slave-m39l"
server_key: "5f5cf8c926c5b0632ba2278b2984dd96"
server_key_iv: "3a74074d3cdbb3354da24fc0e306012e"
server_key_version: "encryptionkey@0"
I20260928 04:54:50.055032 9864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "97c8a0ccec8b428597ae8467849e83d0"
format_stamp: "Formatted at 2026-09-28 04:54:50 on dist-test-slave-m39l"
server_key: "5f5cf8c926c5b0632ba2278b2984dd96"
server_key_iv: "3a74074d3cdbb3354da24fc0e306012e"
server_key_version: "encryptionkey@0"
I20260928 04:54:50.056614 9864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:54:50.057518 9878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:50.057713 9864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:50.057816 9864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "97c8a0ccec8b428597ae8467849e83d0"
format_stamp: "Formatted at 2026-09-28 04:54:50 on dist-test-slave-m39l"
server_key: "5f5cf8c926c5b0632ba2278b2984dd96"
server_key_iv: "3a74074d3cdbb3354da24fc0e306012e"
server_key_version: "encryptionkey@0"
I20260928 04:54:50.057910 9864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:50.068014 9864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:50.068395 9864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:50.068563 9864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:50.068826 9864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:54:50.069187 9864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:54:50.069248 9864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:50.069291 9864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:54:50.069314 9864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:50.075654 9864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40139
I20260928 04:54:50.075690 9991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40139 every 8 connection(s)
I20260928 04:54:50.076144 9864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:54:50.076643 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 9864
I20260928 04:54:50.076766 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:54:50.076977 392 external_mini_cluster.cc:1428] Setting key 7576d2e30cef9a4901880da103aef7bc
I20260928 04:54:50.078118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38505
--builtin_ntp_servers=127.0.98.20:45523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 04:54:50.083117 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:50.077132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37411 (local address 127.0.98.62:38505)
0928 04:54:50.077190 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:50.077192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:50.077484 (+ 292us) server_negotiation.cc:407] Connection header received
0928 04:54:50.078328 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:50.078330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:50.078346 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:50.078420 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:50.079436 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:50.079940 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:50.081312 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:50.081536 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:50.081667 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:50.082782 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:50.082784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:50.082784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:54:50.082791 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:50.082807 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:54:50.082812 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:50.082870 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:50.082987 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 04:54:50.083019 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 04:54:50.083405 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:50.077131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38505 (local address 127.0.98.2:37411)
0928 04:54:50.077336 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:50.077374 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:50.078033 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:50.078537 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:50.078543 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:50.078556 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:50.079303 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:50.079310 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:50.080055 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:50.080057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:50.081178 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:50.081182 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:50.081681 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:50.081682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:50.081863 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:50.082593 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:54:50.082594 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:50.082630 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:54:50.082658 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:50.082878 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:50.082883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:50.082887 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:54:50.082990 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 04:54:50.083126 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260928 04:54:50.083916 9992 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38505
I20260928 04:54:50.084038 9992 heartbeater.cc:464] Registering TS with master...
I20260928 04:54:50.084261 9992 heartbeater.cc:511] Master 127.0.98.62:38505 requested a full tablet report, sending...
I20260928 04:54:50.084730 9673 ts_manager.cc:194] Registered new tserver with Master: 97c8a0ccec8b428597ae8467849e83d0 (127.0.98.2:40139)
I20260928 04:54:50.085304 9673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:37411
W20260928 04:54:50.172760 9995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:50.173022 9995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:50.173050 9995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:50.174558 9995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:50.174620 9995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:50.174652 9995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:54:50.176446 9995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:50.176784 9995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:50.177105 9995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:50.182897 10000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:50.182889 10003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:50.182889 10001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:50.183096 9995 server_base.cc:1034] running on GCE node
I20260928 04:54:50.183529 9995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:50.183807 9995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:50.185000 9995 hybrid_clock.cc:648] HybridClock initialized: now 1790571290184981 us; error 35 us; skew 500 ppm
I20260928 04:54:50.186586 9995 webserver.cc:460] Webserver started at http://127.0.98.3:39177/ using document root <none> and password file <none>
I20260928 04:54:50.186882 9995 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:50.186933 9995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:50.187109 9995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:50.188234 9995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "b53ced982e8e4b3c905068e5dd5b6942"
format_stamp: "Formatted at 2026-09-28 04:54:50 on dist-test-slave-m39l"
server_key: "e52cd3b1cb80dd413130c96329f8827c"
server_key_iv: "85cf17d5a2aa1d09be8937dee1521893"
server_key_version: "encryptionkey@0"
I20260928 04:54:50.188611 9995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b53ced982e8e4b3c905068e5dd5b6942"
format_stamp: "Formatted at 2026-09-28 04:54:50 on dist-test-slave-m39l"
server_key: "e52cd3b1cb80dd413130c96329f8827c"
server_key_iv: "85cf17d5a2aa1d09be8937dee1521893"
server_key_version: "encryptionkey@0"
I20260928 04:54:50.190143 9995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:54:50.191172 10009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:50.191359 9995 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:54:50.191479 9995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "b53ced982e8e4b3c905068e5dd5b6942"
format_stamp: "Formatted at 2026-09-28 04:54:50 on dist-test-slave-m39l"
server_key: "e52cd3b1cb80dd413130c96329f8827c"
server_key_iv: "85cf17d5a2aa1d09be8937dee1521893"
server_key_version: "encryptionkey@0"
I20260928 04:54:50.191571 9995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:50.209949 9995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:50.210353 9995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:50.210600 9995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:50.210897 9995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:54:50.211282 9995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:54:50.211350 9995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:50.211400 9995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:54:50.211426 9995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:50.218649 9995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36969
I20260928 04:54:50.218724 10122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36969 every 8 connection(s)
I20260928 04:54:50.219125 9995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:54:50.224921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 9995
I20260928 04:54:50.225039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:54:50.225238 392 external_mini_cluster.cc:1428] Setting key cf06f99be1aaf76b1b1ae34903d2a856
I20260928 04:54:50.225248 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:50.219973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46569 (local address 127.0.98.62:38505)
0928 04:54:50.220030 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:50.220032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:50.220300 (+ 268us) server_negotiation.cc:407] Connection header received
0928 04:54:50.220878 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:50.220879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:50.220891 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:50.220962 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:50.221687 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:50.222258 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:50.223503 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:50.223691 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:50.223760 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:50.224813 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:50.224815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:50.224815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:54:50.224820 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:50.224835 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:54:50.224838 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:50.224893 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:50.225066 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 04:54:50.225101 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 04:54:50.225485 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:50.219926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38505 (local address 127.0.98.3:46569)
0928 04:54:50.220131 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:50.220176 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:50.220748 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:50.221042 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:50.221046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:50.221055 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:50.221584 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:50.221589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:50.222386 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:50.222388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:50.223329 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:50.223331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:50.223806 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:50.223808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:50.223940 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:50.224607 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:54:50.224608 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:50.224639 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:54:50.224692 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:50.224913 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:50.224920 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:50.224924 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:54:50.225003 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 04:54:50.225138 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260928 04:54:50.226040 10123 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38505
I20260928 04:54:50.226166 10123 heartbeater.cc:464] Registering TS with master...
I20260928 04:54:50.226374 10123 heartbeater.cc:511] Master 127.0.98.62:38505 requested a full tablet report, sending...
I20260928 04:54:50.226845 9673 ts_manager.cc:194] Registered new tserver with Master: b53ced982e8e4b3c905068e5dd5b6942 (127.0.98.3:36969)
I20260928 04:54:50.227258 9673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46569
I20260928 04:54:50.237506 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 04:54:50.239813 9712 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:50.239132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.0.98.62:38505)
0928 04:54:50.239183 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:50.239185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:50.239411 (+ 226us) server_negotiation.cc:407] Connection header received
0928 04:54:50.239491 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:50.239492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:50.239502 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:50.239550 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:50.239723 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33832: BlockingRecv error: recv got EOF from 127.0.0.1:33832 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:54:50.239835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 9733
I20260928 04:54:50.244587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 9864
I20260928 04:54:50.249451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 9995
I20260928 04:54:50.253922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 9642
2026-09-28T04:54:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (642 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T04:54:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:54:50Z Disabled control of system clock
I20260928 04:54:50.281427 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34723
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 04:54:50.366990 10134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:50.367219 10134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:50.367242 10134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:50.368436 10134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:54:50.368480 10134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:50.368492 10134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:50.368505 10134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:54:50.368517 10134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:54:50.369905 10134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34723
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:50.370154 10134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:50.370437 10134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:50.375612 10139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:50.375634 10140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:50.375612 10142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:50.375892 10134 server_base.cc:1034] running on GCE node
I20260928 04:54:50.376258 10134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:50.376605 10134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:50.377848 10134 hybrid_clock.cc:648] HybridClock initialized: now 1790571290377823 us; error 36 us; skew 500 ppm
I20260928 04:54:50.379431 10134 webserver.cc:460] Webserver started at http://127.0.98.62:33415/ using document root <none> and password file <none>
I20260928 04:54:50.379782 10134 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:50.379875 10134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:50.380059 10134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:50.381363 10134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "7c7e7511fab94cbcb237c12a43ba0bf2"
format_stamp: "Formatted at 2026-09-28 04:54:50 on dist-test-slave-m39l"
server_key: "627aaf201581b8145af28f095bd78274"
server_key_iv: "4c40a1249dc3d59b1b0fa4a06ca1b82c"
server_key_version: "encryptionkey@0"
I20260928 04:54:50.381837 10134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "7c7e7511fab94cbcb237c12a43ba0bf2"
format_stamp: "Formatted at 2026-09-28 04:54:50 on dist-test-slave-m39l"
server_key: "627aaf201581b8145af28f095bd78274"
server_key_iv: "4c40a1249dc3d59b1b0fa4a06ca1b82c"
server_key_version: "encryptionkey@0"
I20260928 04:54:50.383536 10134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 04:54:50.384579 10148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:50.384785 10134 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:50.384899 10134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "7c7e7511fab94cbcb237c12a43ba0bf2"
format_stamp: "Formatted at 2026-09-28 04:54:50 on dist-test-slave-m39l"
server_key: "627aaf201581b8145af28f095bd78274"
server_key_iv: "4c40a1249dc3d59b1b0fa4a06ca1b82c"
server_key_version: "encryptionkey@0"
I20260928 04:54:50.385002 10134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:50.388914 10134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:50.389210 10134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:50.389390 10134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:50.393446 10134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34723
I20260928 04:54:50.393509 10200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34723 every 8 connection(s)
I20260928 04:54:50.393874 10134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:54:50.394755 10201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:50.396242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 10134
I20260928 04:54:50.396328 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:54:50.396535 392 external_mini_cluster.cc:1428] Setting key 4850850a3fab923e70d8a52371fda85e
I20260928 04:54:50.399091 10201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2: Bootstrap starting.
I20260928 04:54:50.399595 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:50.397158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38172 (local address 127.0.98.62:34723)
0928 04:54:50.397423 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:50.397433 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:50.397475 (+ 42us) server_negotiation.cc:407] Connection header received
0928 04:54:50.398052 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:50.398062 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:50.398130 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:50.398306 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:50.399025 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:50.399029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:50.399030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:54:50.399035 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:50.399085 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:54:50.399102 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:50.399257 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:50.399292 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 04:54:50.399354 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":83,"threads_started":1}
I20260928 04:54:50.399914 10201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:50.400169 10201 log.cc:824] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2: Log is configured to *not* fsync() on all Append() calls
I20260928 04:54:50.400924 10201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2: No bootstrap required, opened a new log
I20260928 04:54:50.402674 10201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7e7511fab94cbcb237c12a43ba0bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34723 } }
I20260928 04:54:50.402840 10201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:50.402866 10201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7e7511fab94cbcb237c12a43ba0bf2, State: Initialized, Role: FOLLOWER
I20260928 04:54:50.403014 10201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7e7511fab94cbcb237c12a43ba0bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34723 } }
I20260928 04:54:50.403113 10201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:54:50.403141 10201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:54:50.403219 10201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:50.403884 10201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7e7511fab94cbcb237c12a43ba0bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34723 } }
I20260928 04:54:50.404006 10201 leader_election.cc:302] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c7e7511fab94cbcb237c12a43ba0bf2; no voters:
I20260928 04:54:50.404274 10201 leader_election.cc:288] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:54:50.404414 10206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:54:50.404552 10206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2 [term 1 LEADER]: Becoming Leader. State: Replica: 7c7e7511fab94cbcb237c12a43ba0bf2, State: Running, Role: LEADER
I20260928 04:54:50.404593 10201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:54:50.404647 10206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7e7511fab94cbcb237c12a43ba0bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34723 } }
I20260928 04:54:50.405016 10206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c7e7511fab94cbcb237c12a43ba0bf2. Latest consensus state: current_term: 1 leader_uuid: "7c7e7511fab94cbcb237c12a43ba0bf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7e7511fab94cbcb237c12a43ba0bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34723 } } }
I20260928 04:54:50.405025 10207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c7e7511fab94cbcb237c12a43ba0bf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7e7511fab94cbcb237c12a43ba0bf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34723 } } }
I20260928 04:54:50.405117 10206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2 [sys.catalog]: This master's current role is: LEADER
I20260928 04:54:50.405118 10207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2 [sys.catalog]: This master's current role is: LEADER
I20260928 04:54:50.405505 10212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:54:50.406054 10212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:54:50.407927 10212 catalog_manager.cc:1347] Generated new cluster ID: 7fd3941e48b1461bad38e613a26b04eb
I20260928 04:54:50.408097 10212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:54:50.414570 10212 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:54:50.414825 10212 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:54:50.425233 10212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c7e7511fab94cbcb237c12a43ba0bf2: Generated new TSK 0
I20260928 04:54:50.425511 10212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:54:50.428350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34723
--builtin_ntp_servers=127.0.98.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 04:54:50.525588 10225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:50.525882 10225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:50.525925 10225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:50.527442 10225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:50.527535 10225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:50.527586 10225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:54:50.529160 10225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:50.529505 10225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:50.529848 10225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:50.536602 10230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:50.536780 10225 server_base.cc:1034] running on GCE node
W20260928 04:54:50.536626 10231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:50.536952 10233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:50.537214 10225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:50.537550 10225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:50.538741 10225 hybrid_clock.cc:648] HybridClock initialized: now 1790571290538723 us; error 47 us; skew 500 ppm
I20260928 04:54:50.540654 10225 webserver.cc:460] Webserver started at http://127.0.98.1:46179/ using document root <none> and password file <none>
I20260928 04:54:50.541003 10225 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:50.541087 10225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:50.541235 10225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:50.542873 10225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "d4a9f5f586a64dc59c6e7645bef4ae29"
format_stamp: "Formatted at 2026-09-28 04:54:50 on dist-test-slave-m39l"
server_key: "b9e85f43030fa7243bf46e5b4c9a7ea1"
server_key_iv: "4d15d8cfa352d0f447f873d8c67486eb"
server_key_version: "encryptionkey@0"
I20260928 04:54:50.543331 10225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d4a9f5f586a64dc59c6e7645bef4ae29"
format_stamp: "Formatted at 2026-09-28 04:54:50 on dist-test-slave-m39l"
server_key: "b9e85f43030fa7243bf46e5b4c9a7ea1"
server_key_iv: "4d15d8cfa352d0f447f873d8c67486eb"
server_key_version: "encryptionkey@0"
I20260928 04:54:50.545279 10225 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 04:54:50.546414 10239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:50.546681 10225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:50.546823 10225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "d4a9f5f586a64dc59c6e7645bef4ae29"
format_stamp: "Formatted at 2026-09-28 04:54:50 on dist-test-slave-m39l"
server_key: "b9e85f43030fa7243bf46e5b4c9a7ea1"
server_key_iv: "4d15d8cfa352d0f447f873d8c67486eb"
server_key_version: "encryptionkey@0"
I20260928 04:54:50.546934 10225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:50.551851 10225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:50.552246 10225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:50.552448 10225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:50.552727 10225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:54:50.553136 10225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:54:50.553220 10225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:50.553301 10225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:54:50.553355 10225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:50.560914 10225 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37373
I20260928 04:54:50.560959 10352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37373 every 8 connection(s)
I20260928 04:54:50.561408 10225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:54:50.564774 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:50.562299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47573 (local address 127.0.98.62:34723)
0928 04:54:50.562411 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:50.562414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:50.562674 (+ 260us) server_negotiation.cc:407] Connection header received
0928 04:54:50.563155 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:50.563156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:50.563162 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:50.563270 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:50.564462 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:50.564464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:50.564464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:54:50.564475 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:50.564492 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:54:50.564496 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:50.564570 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:50.564679 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 04:54:50.564695 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 04:54:50.564944 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:50.562275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34723 (local address 127.0.98.1:47573)
0928 04:54:50.562516 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:50.562559 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:50.563053 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:50.563345 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:50.563349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:50.563357 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:50.564284 (+ 927us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:54:50.564285 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:50.564318 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:54:50.564348 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:50.564575 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:50.564579 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:50.564581 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:54:50.564640 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 04:54:50.564698 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":116,"threads_started":1}
I20260928 04:54:50.565589 10353 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34723
I20260928 04:54:50.565685 10353 heartbeater.cc:464] Registering TS with master...
I20260928 04:54:50.565876 10353 heartbeater.cc:511] Master 127.0.98.62:34723 requested a full tablet report, sending...
I20260928 04:54:50.566645 10165 ts_manager.cc:194] Registered new tserver with Master: d4a9f5f586a64dc59c6e7645bef4ae29 (127.0.98.1:37373)
I20260928 04:54:50.568251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 10225
I20260928 04:54:50.568324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:54:50.568547 392 external_mini_cluster.cc:1428] Setting key 93c2756929258d0e11de447166b0548b
I20260928 04:54:50.569607 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34723
--builtin_ntp_servers=127.0.98.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 04:54:50.656811 10356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:50.657080 10356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:50.657123 10356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:50.658376 10356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:50.658458 10356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:50.658504 10356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:54:50.660017 10356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:50.660323 10356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:50.660647 10356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:50.666656 10362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:50.666671 10356 server_base.cc:1034] running on GCE node
W20260928 04:54:50.666689 10364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:50.666679 10361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:50.667223 10356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:50.667490 10356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:50.668680 10356 hybrid_clock.cc:648] HybridClock initialized: now 1790571290668664 us; error 29 us; skew 500 ppm
I20260928 04:54:50.670056 10356 webserver.cc:460] Webserver started at http://127.0.98.2:37361/ using document root <none> and password file <none>
I20260928 04:54:50.670385 10356 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:50.670466 10356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:50.670646 10356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:50.671785 10356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "78f272a9aeb64b1ba571bc125133f8c3"
format_stamp: "Formatted at 2026-09-28 04:54:50 on dist-test-slave-m39l"
server_key: "20d57c8226d155693d6bd97001b5ab17"
server_key_iv: "6eb46ed6839e5185c248ca92e9bb8beb"
server_key_version: "encryptionkey@0"
I20260928 04:54:50.672184 10356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "78f272a9aeb64b1ba571bc125133f8c3"
format_stamp: "Formatted at 2026-09-28 04:54:50 on dist-test-slave-m39l"
server_key: "20d57c8226d155693d6bd97001b5ab17"
server_key_iv: "6eb46ed6839e5185c248ca92e9bb8beb"
server_key_version: "encryptionkey@0"
I20260928 04:54:50.673625 10356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:54:50.674726 10370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:50.674974 10356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:50.675091 10356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "78f272a9aeb64b1ba571bc125133f8c3"
format_stamp: "Formatted at 2026-09-28 04:54:50 on dist-test-slave-m39l"
server_key: "20d57c8226d155693d6bd97001b5ab17"
server_key_iv: "6eb46ed6839e5185c248ca92e9bb8beb"
server_key_version: "encryptionkey@0"
I20260928 04:54:50.675179 10356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:50.679059 10356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:50.679374 10356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:50.679529 10356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:50.679746 10356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:54:50.680094 10356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:54:50.680152 10356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:50.680195 10356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:54:50.680223 10356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:50.686450 10356 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43653
I20260928 04:54:50.686551 10483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43653 every 8 connection(s)
I20260928 04:54:50.686908 10356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:54:50.690467 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:50.687771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58663 (local address 127.0.98.62:34723)
0928 04:54:50.687842 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:50.687845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:50.688153 (+ 308us) server_negotiation.cc:407] Connection header received
0928 04:54:50.688718 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:50.688719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:50.688725 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:50.688787 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:50.690168 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:50.690170 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:50.690170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:54:50.690175 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:50.690191 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:54:50.690195 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:50.690263 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:50.690393 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 04:54:50.690406 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 04:54:50.690790 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:50.687775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34723 (local address 127.0.98.2:58663)
0928 04:54:50.688016 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:50.688053 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:50.688577 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:50.688903 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:50.688907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:50.688915 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:50.689960 (+ 1045us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:54:50.689960 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:50.689993 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:54:50.690059 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:50.690322 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:50.690327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:50.690330 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:54:50.690400 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:54:50.690458 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":118,"threads_started":1}
I20260928 04:54:50.691399 10484 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34723
I20260928 04:54:50.691489 10484 heartbeater.cc:464] Registering TS with master...
I20260928 04:54:50.691665 10484 heartbeater.cc:511] Master 127.0.98.62:34723 requested a full tablet report, sending...
I20260928 04:54:50.692096 10165 ts_manager.cc:194] Registered new tserver with Master: 78f272a9aeb64b1ba571bc125133f8c3 (127.0.98.2:43653)
I20260928 04:54:50.694729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 10356
I20260928 04:54:50.694824 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:54:50.695032 392 external_mini_cluster.cc:1428] Setting key 0aff56a80cfb7f431741f35a2b9f813d
I20260928 04:54:50.696056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34723
--builtin_ntp_servers=127.0.98.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 04:54:50.780128 10487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:50.780369 10487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:50.780395 10487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:50.781728 10487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:50.781786 10487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:50.781817 10487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:54:50.783389 10487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:50.783684 10487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:50.784001 10487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:54:50.789666 10487 server_base.cc:1034] running on GCE node
W20260928 04:54:50.789636 10493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:50.789614 10495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:50.789618 10492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:50.790124 10487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:50.790385 10487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:50.791538 10487 hybrid_clock.cc:648] HybridClock initialized: now 1790571290791531 us; error 41 us; skew 500 ppm
I20260928 04:54:50.792749 10487 webserver.cc:460] Webserver started at http://127.0.98.3:39503/ using document root <none> and password file <none>
I20260928 04:54:50.793005 10487 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:50.793075 10487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:50.793220 10487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:50.794328 10487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "e405c57e5ba44e738299bf0fae5a12e6"
format_stamp: "Formatted at 2026-09-28 04:54:50 on dist-test-slave-m39l"
server_key: "e79aa8e4c8a6478841b288093d0e316f"
server_key_iv: "000463535736af17ef750854e5ee7595"
server_key_version: "encryptionkey@0"
I20260928 04:54:50.794740 10487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e405c57e5ba44e738299bf0fae5a12e6"
format_stamp: "Formatted at 2026-09-28 04:54:50 on dist-test-slave-m39l"
server_key: "e79aa8e4c8a6478841b288093d0e316f"
server_key_iv: "000463535736af17ef750854e5ee7595"
server_key_version: "encryptionkey@0"
I20260928 04:54:50.796191 10487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:54:50.797003 10501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:50.797183 10487 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:54:50.797282 10487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "e405c57e5ba44e738299bf0fae5a12e6"
format_stamp: "Formatted at 2026-09-28 04:54:50 on dist-test-slave-m39l"
server_key: "e79aa8e4c8a6478841b288093d0e316f"
server_key_iv: "000463535736af17ef750854e5ee7595"
server_key_version: "encryptionkey@0"
I20260928 04:54:50.797412 10487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:50.800865 10487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:50.801146 10487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:50.801290 10487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:50.801505 10487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:54:50.801833 10487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:54:50.801898 10487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:50.801937 10487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:54:50.801965 10487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:50.808447 10487 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40951
I20260928 04:54:50.808480 10614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40951 every 8 connection(s)
I20260928 04:54:50.808833 10487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:54:50.811148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 10487
I20260928 04:54:50.811264 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:54:50.811508 392 external_mini_cluster.cc:1428] Setting key cdb082cee28c6da26b98a22317241b45
I20260928 04:54:50.812377 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:50.809588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48143 (local address 127.0.98.62:34723)
0928 04:54:50.809634 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:50.809637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:50.809878 (+ 241us) server_negotiation.cc:407] Connection header received
0928 04:54:50.810406 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:50.810407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:50.810413 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:50.810454 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:50.812085 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:50.812086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:50.812086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:54:50.812093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:50.812110 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:54:50.812115 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:50.812178 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:50.812277 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 04:54:50.812295 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:54:50.812577 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:50.809554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34723 (local address 127.0.98.3:48143)
0928 04:54:50.809745 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:50.809783 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:50.810288 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:50.810572 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:50.810576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:50.810587 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:50.811743 (+ 1156us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:54:50.811744 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:50.811805 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:54:50.811844 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:50.812186 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:50.812192 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:50.812195 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:54:50.812273 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 04:54:50.812343 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260928 04:54:50.813047 10615 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34723
I20260928 04:54:50.813143 10615 heartbeater.cc:464] Registering TS with master...
I20260928 04:54:50.813280 10615 heartbeater.cc:511] Master 127.0.98.62:34723 requested a full tablet report, sending...
I20260928 04:54:50.813612 10165 ts_manager.cc:194] Registered new tserver with Master: e405c57e5ba44e738299bf0fae5a12e6 (127.0.98.3:40951)
I20260928 04:54:50.823688 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 04:54:50.826028 10204 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:54:50.825411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.0.98.62:34723)
0928 04:54:50.825527 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:50.825529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:50.825628 (+ 99us) server_negotiation.cc:407] Connection header received
0928 04:54:50.825751 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:50.825752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:50.825757 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:50.825828 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:50.825970 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38180: BlockingRecv error: recv got EOF from 127.0.0.1:38180 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 04:54:50.826303 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 10225
I20260928 04:54:50.830768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 10356
I20260928 04:54:50.835317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 10487
I20260928 04:54:50.839872 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 10134
2026-09-28T04:54:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (586 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:54:50 dist-test-slave-m39l krb5kdc[10625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:54:50 dist-test-slave-m39l krb5kdc[10625](info): set up 2 sockets
Sep 28 04:54:50 dist-test-slave-m39l krb5kdc[10625](info): commencing operation
krb5kdc: starting...
W20260928 04:54:51.883002 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:54:51 dist-test-slave-m39l krb5kdc[10625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:54:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:54:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:54:52.015682 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39939
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:54:52.102319 10638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:52.102653 10638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:52.102696 10638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:52.103945 10638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:54:52.104033 10638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:52.104058 10638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:52.104090 10638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:54:52.104115 10638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:54:52.105731 10638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:52.106048 10638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:52.106300 10638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:52.111631 10646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:52.111668 10644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:52.111747 10638 server_base.cc:1034] running on GCE node
W20260928 04:54:52.111825 10643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:52.112135 10638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:52.112418 10638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:52.113567 10638 hybrid_clock.cc:648] HybridClock initialized: now 1790571292113553 us; error 48 us; skew 500 ppm
Sep 28 04:54:52 dist-test-slave-m39l krb5kdc[10625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:54:52.116146 10638 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:54:52.116696 10638 webserver.cc:460] Webserver started at http://127.0.98.62:37467/ using document root <none> and password file <none>
I20260928 04:54:52.116891 10638 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:52.116933 10638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:52.117025 10638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:52.118085 10638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "c9cc5b3d02a040febf228e0948d96880"
format_stamp: "Formatted at 2026-09-28 04:54:52 on dist-test-slave-m39l"
server_key: "c70c828129cb9c1a5a3653adb2e7c821"
server_key_iv: "752a4da4c6a7aeab678c712d2c2e71b1"
server_key_version: "encryptionkey@0"
I20260928 04:54:52.118408 10638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "c9cc5b3d02a040febf228e0948d96880"
format_stamp: "Formatted at 2026-09-28 04:54:52 on dist-test-slave-m39l"
server_key: "c70c828129cb9c1a5a3653adb2e7c821"
server_key_iv: "752a4da4c6a7aeab678c712d2c2e71b1"
server_key_version: "encryptionkey@0"
I20260928 04:54:52.119849 10638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:52.120644 10653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:52.120837 10638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:52.120950 10638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "c9cc5b3d02a040febf228e0948d96880"
format_stamp: "Formatted at 2026-09-28 04:54:52 on dist-test-slave-m39l"
server_key: "c70c828129cb9c1a5a3653adb2e7c821"
server_key_iv: "752a4da4c6a7aeab678c712d2c2e71b1"
server_key_version: "encryptionkey@0"
I20260928 04:54:52.121042 10638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:52.135924 10638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:52.138161 10638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:52.138350 10638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:52.142639 10638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39939
I20260928 04:54:52.142673 10705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39939 every 8 connection(s)
I20260928 04:54:52.143047 10638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:54:52.144063 10706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:52.146481 10706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880: Bootstrap starting.
I20260928 04:54:52.147202 10706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:52.147521 10706 log.cc:824] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880: Log is configured to *not* fsync() on all Append() calls
I20260928 04:54:52.148428 10706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880: No bootstrap required, opened a new log
I20260928 04:54:52.150050 10706 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9cc5b3d02a040febf228e0948d96880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39939 } }
I20260928 04:54:52.150198 10706 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:52.150221 10706 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9cc5b3d02a040febf228e0948d96880, State: Initialized, Role: FOLLOWER
I20260928 04:54:52.150410 10706 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9cc5b3d02a040febf228e0948d96880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39939 } }
I20260928 04:54:52.150507 10706 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:54:52.150553 10706 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:54:52.150599 10706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:52.151129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 10638
I20260928 04:54:52.151229 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:54:52.151422 392 external_mini_cluster.cc:1428] Setting key ed26a8ab03e1b630701c798798cde20b
I20260928 04:54:52.151710 10706 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9cc5b3d02a040febf228e0948d96880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39939 } }
I20260928 04:54:52.151840 10706 leader_election.cc:302] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9cc5b3d02a040febf228e0948d96880; no voters:
I20260928 04:54:52.152019 10706 leader_election.cc:288] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:54:52.152103 10710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:54:52.152252 10710 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880 [term 1 LEADER]: Becoming Leader. State: Replica: c9cc5b3d02a040febf228e0948d96880, State: Running, Role: LEADER
I20260928 04:54:52.152376 10706 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:54:52.152375 10710 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9cc5b3d02a040febf228e0948d96880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39939 } }
I20260928 04:54:52.153846 10713 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9cc5b3d02a040febf228e0948d96880. Latest consensus state: current_term: 1 leader_uuid: "c9cc5b3d02a040febf228e0948d96880" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9cc5b3d02a040febf228e0948d96880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39939 } } }
I20260928 04:54:52.153940 10713 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880 [sys.catalog]: This master's current role is: LEADER
W20260928 04:54:52.154325 10725 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 04:54:52.153865 10712 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9cc5b3d02a040febf228e0948d96880" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9cc5b3d02a040febf228e0948d96880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39939 } } }
W20260928 04:54:52.154424 10725 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:54:52.154443 10712 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880 [sys.catalog]: This master's current role is: LEADER
I20260928 04:54:52.154505 10726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:54:52.154652 10726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:54:52.156157 10726 catalog_manager.cc:1347] Generated new cluster ID: 64f7f1c71ca94302bfa9433ce0fa0e4c
I20260928 04:54:52.156215 10726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:54:52 dist-test-slave-m39l krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:54:52.162837 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.151908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33452 (local address 127.0.98.62:39939)
0928 04:54:52.152085 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:52.152097 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:52.152140 (+ 43us) server_negotiation.cc:407] Connection header received
0928 04:54:52.152348 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:52.152356 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:52.152416 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:52.152552 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:52.153262 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.154806 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.155879 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.156437 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.156522 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.159131 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:52.159136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:52.159137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:52.159149 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:52.161226 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:52.161628 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:52.161631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:52.161631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:52.161740 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:52.161931 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:52.161932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:52.161932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:52.162238 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:52.162360 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:52.162464 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 04:54:52.162616 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260928 04:54:52.173619 10726 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:54:52.174399 10726 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:54:52.180186 10726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880: Generated new TSK 0
I20260928 04:54:52.180413 10726 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:54:52.215404 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39939
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:54:52.301937 10732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:52.302177 10732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:52.302210 10732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:52.303488 10732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:52.303540 10732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:52.303570 10732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:54:52.305225 10732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:52.305505 10732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:52.305779 10732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:52.311160 10740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:52.311251 10737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:52.311216 10738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:52.311352 10732 server_base.cc:1034] running on GCE node
I20260928 04:54:52.311650 10732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:52.311940 10732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:52.313131 10732 hybrid_clock.cc:648] HybridClock initialized: now 1790571292313099 us; error 81 us; skew 500 ppm
Sep 28 04:54:52 dist-test-slave-m39l krb5kdc[10625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:54:52.316155 10732 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:54:52.316704 10732 webserver.cc:460] Webserver started at http://127.0.98.1:39017/ using document root <none> and password file <none>
I20260928 04:54:52.316990 10732 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:52.317068 10732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:52.317217 10732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:52.318432 10732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "90bab08e8aad40329a5f2e98aa4b5721"
format_stamp: "Formatted at 2026-09-28 04:54:52 on dist-test-slave-m39l"
server_key: "d76c37e48a6bf274876acd4e9051b76b"
server_key_iv: "d13400d819c36678df48c45c34a5ee5b"
server_key_version: "encryptionkey@0"
I20260928 04:54:52.318933 10732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "90bab08e8aad40329a5f2e98aa4b5721"
format_stamp: "Formatted at 2026-09-28 04:54:52 on dist-test-slave-m39l"
server_key: "d76c37e48a6bf274876acd4e9051b76b"
server_key_iv: "d13400d819c36678df48c45c34a5ee5b"
server_key_version: "encryptionkey@0"
I20260928 04:54:52.320533 10732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:54:52.321491 10747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:52.321715 10732 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:52.321831 10732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "90bab08e8aad40329a5f2e98aa4b5721"
format_stamp: "Formatted at 2026-09-28 04:54:52 on dist-test-slave-m39l"
server_key: "d76c37e48a6bf274876acd4e9051b76b"
server_key_iv: "d13400d819c36678df48c45c34a5ee5b"
server_key_version: "encryptionkey@0"
I20260928 04:54:52.321930 10732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:52.360605 10732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:52.363185 10732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:52.363409 10732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:52.363686 10732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:54:52.364048 10732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:54:52.364114 10732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:52.364162 10732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:54:52.364192 10732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:52.370885 10732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45491
I20260928 04:54:52.370929 10860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45491 every 8 connection(s)
I20260928 04:54:52.371341 10732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:54:52 dist-test-slave-m39l krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:54:52.381345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 10732
I20260928 04:54:52.381417 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:54:52.381600 392 external_mini_cluster.cc:1428] Setting key fd461dcea041d85ead40e764ba7b9d41
I20260928 04:54:52.382251 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.372195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54751 (local address 127.0.98.62:39939)
0928 04:54:52.372240 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:52.372242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:52.372486 (+ 244us) server_negotiation.cc:407] Connection header received
0928 04:54:52.373014 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:52.373016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:52.373031 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:52.373085 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:52.374180 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.374840 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.375961 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.376163 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.376292 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.378946 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:52.378948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:52.378948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:52.378956 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:52.380605 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:52.381094 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:52.381096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:52.381096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:52.381161 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:52.381507 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:52.381508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:52.381509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:52.381656 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:52.381721 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:52.382060 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 04:54:52.382150 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:54:52.382539 10863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.372182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39939 (local address 127.0.98.1:54751)
0928 04:54:52.372353 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:52.372393 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:52.372911 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:52.373147 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:52.373151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:52.373801 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:52.374023 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:52.374029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.374958 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.374959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:52.375822 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:52.375855 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.376324 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.376325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:52.376455 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.377153 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:52.377162 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:52.378834 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:52.380721 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:52.380731 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:52.380734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:52.380995 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:52.381236 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:52.381237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:52.381237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:52.381409 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:52.381713 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:52.381720 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:52.381906 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 04:54:52.382037 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 04:54:52.382224 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260928 04:54:52.382987 10861 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39939
I20260928 04:54:52.383085 10861 heartbeater.cc:464] Registering TS with master...
I20260928 04:54:52.383297 10861 heartbeater.cc:511] Master 127.0.98.62:39939 requested a full tablet report, sending...
I20260928 04:54:52.383816 10670 ts_manager.cc:194] Registered new tserver with Master: 90bab08e8aad40329a5f2e98aa4b5721 (127.0.98.1:45491)
I20260928 04:54:52.384459 10670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54751
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:54:52.410424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39939
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:54:52.493785 10866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:52.494084 10866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:52.494127 10866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:52.495546 10866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:52.495630 10866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:52.495677 10866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:54:52.497268 10866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:52.497576 10866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:52.497840 10866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:52.503455 10872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:52.503427 10871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:52.503515 10866 server_base.cc:1034] running on GCE node
W20260928 04:54:52.503443 10874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:52.503831 10866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:52.504055 10866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:52.505234 10866 hybrid_clock.cc:648] HybridClock initialized: now 1790571292505220 us; error 63 us; skew 500 ppm
Sep 28 04:54:52 dist-test-slave-m39l krb5kdc[10625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:54:52.507942 10866 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:54:52.508361 10866 webserver.cc:460] Webserver started at http://127.0.98.2:40289/ using document root <none> and password file <none>
I20260928 04:54:52.508551 10866 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:52.508594 10866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:52.508692 10866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:52.509706 10866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "373e2c6f54c44deab75945d0dadebea6"
format_stamp: "Formatted at 2026-09-28 04:54:52 on dist-test-slave-m39l"
server_key: "636753618be83eff9c6b691153dee992"
server_key_iv: "bb4af9dada5465711a8dbf1604127382"
server_key_version: "encryptionkey@0"
I20260928 04:54:52.510049 10866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "373e2c6f54c44deab75945d0dadebea6"
format_stamp: "Formatted at 2026-09-28 04:54:52 on dist-test-slave-m39l"
server_key: "636753618be83eff9c6b691153dee992"
server_key_iv: "bb4af9dada5465711a8dbf1604127382"
server_key_version: "encryptionkey@0"
I20260928 04:54:52.511515 10866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:54:52.512413 10881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:52.512630 10866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:52.512720 10866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "373e2c6f54c44deab75945d0dadebea6"
format_stamp: "Formatted at 2026-09-28 04:54:52 on dist-test-slave-m39l"
server_key: "636753618be83eff9c6b691153dee992"
server_key_iv: "bb4af9dada5465711a8dbf1604127382"
server_key_version: "encryptionkey@0"
I20260928 04:54:52.512825 10866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:52.531018 10866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:52.533437 10866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:52.533627 10866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:52.533924 10866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:54:52.534256 10866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:54:52.534293 10866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:52.534323 10866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:54:52.534348 10866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:52.540361 10866 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38471
I20260928 04:54:52.540455 10994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38471 every 8 connection(s)
I20260928 04:54:52.540814 10866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:54:52.545997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 10866
I20260928 04:54:52.546139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:54:52.546383 392 external_mini_cluster.cc:1428] Setting key 494d794ba1c214d5b641433b79f4c3b8
Sep 28 04:54:52 dist-test-slave-m39l krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:54:52.552340 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.541805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46679 (local address 127.0.98.62:39939)
0928 04:54:52.541879 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:52.541882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:52.542097 (+ 215us) server_negotiation.cc:407] Connection header received
0928 04:54:52.542725 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:52.542727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:52.542738 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:52.542805 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:52.543941 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.544489 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.545577 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.545774 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.545840 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.548843 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:52.548845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:52.548846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:52.548854 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:52.550375 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:52.551069 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:52.551071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:52.551071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:52.551150 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:52.551485 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:52.551486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:52.551487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:52.551627 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:52.551723 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:52.552126 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 04:54:52.552223 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 04:54:52.552599 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.541773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39939 (local address 127.0.98.2:46679)
0928 04:54:52.541948 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:52.541994 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:52.542597 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:52.542870 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:52.542873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:52.543545 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:52.543779 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:52.543785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.544606 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.544607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:52.545481 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:52.545484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.545898 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.545899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:52.546067 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.546948 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:52.546957 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:52.548720 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:52.550630 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:52.550641 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:52.550644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:52.550975 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:52.551281 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:52.551282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:52.551282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:52.551402 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:52.551711 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:52.551718 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:52.551975 (+ 257us) client_negotiation.cc:769] Sending connection context
0928 04:54:52.552104 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 04:54:52.552324 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260928 04:54:52.553102 10995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39939
I20260928 04:54:52.553227 10995 heartbeater.cc:464] Registering TS with master...
I20260928 04:54:52.553453 10995 heartbeater.cc:511] Master 127.0.98.62:39939 requested a full tablet report, sending...
I20260928 04:54:52.553949 10670 ts_manager.cc:194] Registered new tserver with Master: 373e2c6f54c44deab75945d0dadebea6 (127.0.98.2:38471)
I20260928 04:54:52.554405 10670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46679
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:54:52.578693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39939
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:54:52.667419 11000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:54:52.667670 11000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:54:52.667699 11000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:54:52.668849 11000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:54:52.668900 11000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:54:52.668928 11000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:54:52.670400 11000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:54:52.670715 11000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:54:52.670936 11000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:54:52.676455 11008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:52.676484 11006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:54:52.676481 11005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:54:52.676774 11000 server_base.cc:1034] running on GCE node
I20260928 04:54:52.677007 11000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:54:52.677280 11000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:54:52.678438 11000 hybrid_clock.cc:648] HybridClock initialized: now 1790571292678426 us; error 34 us; skew 500 ppm
Sep 28 04:54:52 dist-test-slave-m39l krb5kdc[10625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:54:52.681587 11000 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:54:52.682130 11000 webserver.cc:460] Webserver started at http://127.0.98.3:35389/ using document root <none> and password file <none>
I20260928 04:54:52.682396 11000 fs_manager.cc:359] Metadata directory not provided
I20260928 04:54:52.682446 11000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:54:52.682615 11000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:54:52.683765 11000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "22a7236217074a7a9abc3133282d5d64"
format_stamp: "Formatted at 2026-09-28 04:54:52 on dist-test-slave-m39l"
server_key: "b4bbbc17c2ab11c82ff44be4854a576c"
server_key_iv: "7e50d7d7a8bef7765f40ff2493cea0ac"
server_key_version: "encryptionkey@0"
I20260928 04:54:52.684135 11000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "22a7236217074a7a9abc3133282d5d64"
format_stamp: "Formatted at 2026-09-28 04:54:52 on dist-test-slave-m39l"
server_key: "b4bbbc17c2ab11c82ff44be4854a576c"
server_key_iv: "7e50d7d7a8bef7765f40ff2493cea0ac"
server_key_version: "encryptionkey@0"
I20260928 04:54:52.685555 11000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:54:52.686444 11015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:52.686709 11000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:54:52.686837 11000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "22a7236217074a7a9abc3133282d5d64"
format_stamp: "Formatted at 2026-09-28 04:54:52 on dist-test-slave-m39l"
server_key: "b4bbbc17c2ab11c82ff44be4854a576c"
server_key_iv: "7e50d7d7a8bef7765f40ff2493cea0ac"
server_key_version: "encryptionkey@0"
I20260928 04:54:52.686923 11000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:54:52.704327 11000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:54:52.706952 11000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:54:52.707136 11000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:54:52.707392 11000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:54:52.707721 11000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:54:52.707785 11000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:52.707828 11000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:54:52.707852 11000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:54:52.714092 11000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39463
I20260928 04:54:52.714217 11128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39463 every 8 connection(s)
I20260928 04:54:52.714542 11000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:54:52 dist-test-slave-m39l krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:54:52.724433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 11000
I20260928 04:54:52.724538 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:54:52.724747 392 external_mini_cluster.cc:1428] Setting key 9e91963de8813be205de61ceaf607d46
I20260928 04:54:52.725097 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.715298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34313 (local address 127.0.98.62:39939)
0928 04:54:52.715348 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:52.715349 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:52.715576 (+ 227us) server_negotiation.cc:407] Connection header received
0928 04:54:52.716215 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:52.716217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:52.716235 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:52.716299 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:52.717406 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.717963 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.719208 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.719420 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.719479 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.721777 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:52.721779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:52.721780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:52.721789 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:52.723311 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:52.723836 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:52.723837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:52.723838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:52.723904 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:52.724167 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:52.724168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:52.724169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:52.724348 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:52.724407 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:52.724856 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 04:54:52.724950 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:54:52.725298 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.715318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39939 (local address 127.0.98.3:34313)
0928 04:54:52.715458 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:52.715496 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:52.716045 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:52.716361 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:52.716365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:52.717051 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:54:52.717295 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:52.717301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.718102 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.718103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:52.719120 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:52.719123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.719502 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.719503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:52.719605 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.720123 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:54:52.720132 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:54:52.721664 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:54:52.723436 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:52.723446 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:52.723448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:52.723702 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:52.723978 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:54:52.723979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:54:52.723979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:54:52.724106 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:54:52.724422 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:54:52.724429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:54:52.724684 (+ 255us) client_negotiation.cc:769] Sending connection context
0928 04:54:52.724810 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 04:54:52.725035 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260928 04:54:52.725795 11129 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39939
I20260928 04:54:52.725910 11129 heartbeater.cc:464] Registering TS with master...
I20260928 04:54:52.726066 11129 heartbeater.cc:511] Master 127.0.98.62:39939 requested a full tablet report, sending...
I20260928 04:54:52.726456 10670 ts_manager.cc:194] Registered new tserver with Master: 22a7236217074a7a9abc3133282d5d64 (127.0.98.3:39463)
I20260928 04:54:52.726897 10670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34313
I20260928 04:54:52.737103 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:54:52.745262 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.738668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.0.98.62:39939)
0928 04:54:52.738737 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:52.738740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:52.738776 (+ 36us) server_negotiation.cc:407] Connection header received
0928 04:54:52.738869 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:52.738870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:52.738881 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:52.738953 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:54:52.739597 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.740142 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.740917 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.741107 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.741197 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.742551 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:54:52.742552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:54:52.742553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:54:52.742560 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:54:52.743965 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:52.744365 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:52.744367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:52.744367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:52.744434 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:54:52.744633 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:54:52.744634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:54:52.744635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:54:52.744801 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:54:52.744909 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:52.745055 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 04:54:52.745143 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 04:54:52.747350 10670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:54:52.749224 10670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:54:52.757527 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.752675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35898 (local address 127.0.98.2:38471)
0928 04:54:52.752868 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:52.752873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:52.753150 (+ 277us) server_negotiation.cc:407] Connection header received
0928 04:54:52.753292 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:52.753295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:52.753312 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:52.753441 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:54:52.753736 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.754685 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.756161 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.756958 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.757041 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.757109 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:52.757383 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 04:54:52.757432 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
I20260928 04:54:52.757540 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.752748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45491 (local address 127.0.0.1:40828)
0928 04:54:52.753321 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:52.753331 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:52.753389 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:52.753668 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:52.753669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:52.753672 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:54:52.753709 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:52.753711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.755142 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.755143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:52.756189 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:52.756191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.757152 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.757153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:52.757249 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.757265 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 04:54:52.757348 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 04:54:52.757429 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":26,"thread_start_us":86,"threads_started":1}
I20260928 04:54:52.757946 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.752479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38471 (local address 127.0.0.1:35898)
0928 04:54:52.753065 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:52.753077 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:52.753183 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:52.753540 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:52.753543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:52.753545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:54:52.753622 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:52.753623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.754790 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.754791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:52.756056 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:52.756058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.757153 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.757154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:52.757260 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.757265 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:54:52.757348 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 04:54:52.757429 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":9,"thread_start_us":88,"threads_started":1}
I20260928 04:54:52.757966 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.752875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.0.98.1:45491)
0928 04:54:52.753074 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:52.753080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:52.753390 (+ 310us) server_negotiation.cc:407] Connection header received
0928 04:54:52.753558 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:52.753561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:52.753577 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:52.753718 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:54:52.753792 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.755028 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.756315 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.756989 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.757059 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.757112 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:52.757811 (+ 699us) server_negotiation.cc:299] Negotiation successful
0928 04:54:52.757877 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20260928 04:54:52.758214 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.752479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39463 (local address 127.0.0.1:41240)
0928 04:54:52.753001 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:52.753062 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:52.753154 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:52.753775 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:52.753776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:52.753778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:54:52.753812 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:52.753813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.756273 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.756273 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:52.756956 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:52.756958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.757937 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.757938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:52.758025 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.758030 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:54:52.758097 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 04:54:52.758144 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":92,"threads_started":1}
I20260928 04:54:52.758241 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.752641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.0.98.3:39463)
0928 04:54:52.752945 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:52.752948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:52.753138 (+ 190us) server_negotiation.cc:407] Connection header received
0928 04:54:52.753558 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:52.753561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:52.753577 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:52.753695 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:54:52.753896 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.755027 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.757065 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.757868 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.757937 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.757998 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:52.758105 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 04:54:52.758144 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":86,"threads_started":1}
I20260928 04:54:52.758792 10929 tablet_service.cc:1467] Processing CreateTablet for tablet 68fa61b0e6a54f298585c3a4083611f1 (DEFAULT_TABLE table=test-table [id=38df6ac4591f4e89b0d2e6a3a15a03bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:54:52.759002 11063 tablet_service.cc:1467] Processing CreateTablet for tablet 68fa61b0e6a54f298585c3a4083611f1 (DEFAULT_TABLE table=test-table [id=38df6ac4591f4e89b0d2e6a3a15a03bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:54:52.758950 10795 tablet_service.cc:1467] Processing CreateTablet for tablet 68fa61b0e6a54f298585c3a4083611f1 (DEFAULT_TABLE table=test-table [id=38df6ac4591f4e89b0d2e6a3a15a03bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:54:52.759182 10929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68fa61b0e6a54f298585c3a4083611f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:52.759307 11063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68fa61b0e6a54f298585c3a4083611f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:52.759382 10795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68fa61b0e6a54f298585c3a4083611f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:54:52.762053 11146 tablet_bootstrap.cc:492] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721: Bootstrap starting.
I20260928 04:54:52.762602 11148 tablet_bootstrap.cc:492] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6: Bootstrap starting.
I20260928 04:54:52.763052 11146 tablet_bootstrap.cc:654] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:52.763248 11148 tablet_bootstrap.cc:654] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:52.763675 11146 log.cc:824] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721: Log is configured to *not* fsync() on all Append() calls
I20260928 04:54:52.763844 11148 log.cc:824] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6: Log is configured to *not* fsync() on all Append() calls
I20260928 04:54:52.764191 11147 tablet_bootstrap.cc:492] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64: Bootstrap starting.
I20260928 04:54:52.764490 11146 tablet_bootstrap.cc:492] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721: No bootstrap required, opened a new log
I20260928 04:54:52.764569 11146 ts_tablet_manager.cc:1397] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 04:54:52.764828 11147 tablet_bootstrap.cc:654] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64: Neither blocks nor log segments found. Creating new log.
I20260928 04:54:52.765413 11147 log.cc:824] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64: Log is configured to *not* fsync() on all Append() calls
I20260928 04:54:52.766242 11147 tablet_bootstrap.cc:492] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64: No bootstrap required, opened a new log
I20260928 04:54:52.766311 11147 ts_tablet_manager.cc:1397] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:54:52.766790 11146 raft_consensus.cc:348] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a7236217074a7a9abc3133282d5d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39463 } } peers { permanent_uuid: "90bab08e8aad40329a5f2e98aa4b5721" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45491 } } peers { permanent_uuid: "373e2c6f54c44deab75945d0dadebea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38471 } }
I20260928 04:54:52.766987 11146 raft_consensus.cc:374] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:52.767023 11146 raft_consensus.cc:729] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90bab08e8aad40329a5f2e98aa4b5721, State: Initialized, Role: FOLLOWER
I20260928 04:54:52.767169 11146 consensus_queue.cc:260] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a7236217074a7a9abc3133282d5d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39463 } } peers { permanent_uuid: "90bab08e8aad40329a5f2e98aa4b5721" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45491 } } peers { permanent_uuid: "373e2c6f54c44deab75945d0dadebea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38471 } }
I20260928 04:54:52.767485 11146 ts_tablet_manager.cc:1428] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 04:54:52.767527 10861 heartbeater.cc:503] Master 127.0.98.62:39939 was elected leader, sending a full tablet report...
I20260928 04:54:52.768129 11148 tablet_bootstrap.cc:492] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6: No bootstrap required, opened a new log
I20260928 04:54:52.768224 11148 ts_tablet_manager.cc:1397] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 04:54:52.768508 11147 raft_consensus.cc:348] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a7236217074a7a9abc3133282d5d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39463 } } peers { permanent_uuid: "90bab08e8aad40329a5f2e98aa4b5721" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45491 } } peers { permanent_uuid: "373e2c6f54c44deab75945d0dadebea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38471 } }
I20260928 04:54:52.768690 11147 raft_consensus.cc:374] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:52.768738 11147 raft_consensus.cc:729] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a7236217074a7a9abc3133282d5d64, State: Initialized, Role: FOLLOWER
I20260928 04:54:52.768886 11147 consensus_queue.cc:260] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a7236217074a7a9abc3133282d5d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39463 } } peers { permanent_uuid: "90bab08e8aad40329a5f2e98aa4b5721" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45491 } } peers { permanent_uuid: "373e2c6f54c44deab75945d0dadebea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38471 } }
I20260928 04:54:52.769201 11147 ts_tablet_manager.cc:1428] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:54:52.769232 11129 heartbeater.cc:503] Master 127.0.98.62:39939 was elected leader, sending a full tablet report...
I20260928 04:54:52.770354 11148 raft_consensus.cc:348] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a7236217074a7a9abc3133282d5d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39463 } } peers { permanent_uuid: "90bab08e8aad40329a5f2e98aa4b5721" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45491 } } peers { permanent_uuid: "373e2c6f54c44deab75945d0dadebea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38471 } }
I20260928 04:54:52.770579 11148 raft_consensus.cc:374] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:54:52.770634 11148 raft_consensus.cc:729] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 373e2c6f54c44deab75945d0dadebea6, State: Initialized, Role: FOLLOWER
I20260928 04:54:52.770767 11148 consensus_queue.cc:260] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a7236217074a7a9abc3133282d5d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39463 } } peers { permanent_uuid: "90bab08e8aad40329a5f2e98aa4b5721" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45491 } } peers { permanent_uuid: "373e2c6f54c44deab75945d0dadebea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38471 } }
I20260928 04:54:52.771090 11148 ts_tablet_manager.cc:1428] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:54:52.771122 10995 heartbeater.cc:503] Master 127.0.98.62:39939 was elected leader, sending a full tablet report...
W20260928 04:54:52.791491 10996 tablet.cc:2369] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:54:52.872133 10862 tablet.cc:2369] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:54:52.927043 11152 raft_consensus.cc:482] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:54:52.927189 11152 raft_consensus.cc:504] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a7236217074a7a9abc3133282d5d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39463 } } peers { permanent_uuid: "90bab08e8aad40329a5f2e98aa4b5721" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45491 } } peers { permanent_uuid: "373e2c6f54c44deab75945d0dadebea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38471 } }
I20260928 04:54:52.927556 11152 leader_election.cc:288] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22a7236217074a7a9abc3133282d5d64 (127.0.98.3:39463), 373e2c6f54c44deab75945d0dadebea6 (127.0.98.2:38471)
I20260928 04:54:52.930850 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.927898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51257 (local address 127.0.98.3:39463)
0928 04:54:52.927952 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:52.927954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:52.928044 (+ 90us) server_negotiation.cc:407] Connection header received
0928 04:54:52.928117 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:52.928118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:52.928134 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:52.928209 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:54:52.928406 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.928862 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.930039 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.930410 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.930470 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.930479 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:52.930707 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 04:54:52.930761 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 04:54:52.930881 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.927847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55699 (local address 127.0.98.2:38471)
0928 04:54:52.927901 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:52.927904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:52.928044 (+ 140us) server_negotiation.cc:407] Connection header received
0928 04:54:52.928103 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:52.928104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:52.928119 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:52.928179 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:54:52.928359 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.928939 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.929976 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.930425 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.930569 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.930580 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:52.930762 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 04:54:52.930803 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 04:54:52.930958 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.927826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39463 (local address 127.0.98.1:51257)
0928 04:54:52.927956 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:52.927967 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:52.928039 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:52.928234 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:52.928235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:52.928237 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:54:52.928331 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:52.928334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.928954 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.928956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:52.929937 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:52.929940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.930534 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.930536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:52.930617 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.930622 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:54:52.930678 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 04:54:52.930761 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":32,"thread_start_us":67,"threads_started":1}
I20260928 04:54:52.930985 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.927770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38471 (local address 127.0.98.1:55699)
0928 04:54:52.927944 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:52.927955 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:54:52.928041 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:54:52.928201 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:54:52.928202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:54:52.928205 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:54:52.928300 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:52.928302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.929031 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.929032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:52.929872 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:54:52.929875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.930578 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.930579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:54:52.930647 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.930651 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:54:52.930730 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 04:54:52.930775 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":86,"threads_started":1}
I20260928 04:54:52.931383 11083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68fa61b0e6a54f298585c3a4083611f1" candidate_uuid: "90bab08e8aad40329a5f2e98aa4b5721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a7236217074a7a9abc3133282d5d64" is_pre_election: true
I20260928 04:54:52.931568 11083 raft_consensus.cc:2454] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90bab08e8aad40329a5f2e98aa4b5721 in term 0.
I20260928 04:54:52.931792 10949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68fa61b0e6a54f298585c3a4083611f1" candidate_uuid: "90bab08e8aad40329a5f2e98aa4b5721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "373e2c6f54c44deab75945d0dadebea6" is_pre_election: true
I20260928 04:54:52.931869 10748 leader_election.cc:302] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a7236217074a7a9abc3133282d5d64, 90bab08e8aad40329a5f2e98aa4b5721; no voters:
I20260928 04:54:52.931996 10949 raft_consensus.cc:2454] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90bab08e8aad40329a5f2e98aa4b5721 in term 0.
I20260928 04:54:52.932157 11152 raft_consensus.cc:2781] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:54:52.932235 11152 raft_consensus.cc:482] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:54:52.932269 11152 raft_consensus.cc:3037] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:52.933036 11152 raft_consensus.cc:504] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a7236217074a7a9abc3133282d5d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39463 } } peers { permanent_uuid: "90bab08e8aad40329a5f2e98aa4b5721" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45491 } } peers { permanent_uuid: "373e2c6f54c44deab75945d0dadebea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38471 } }
I20260928 04:54:52.933202 11152 leader_election.cc:288] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [CANDIDATE]: Term 1 election: Requested vote from peers 22a7236217074a7a9abc3133282d5d64 (127.0.98.3:39463), 373e2c6f54c44deab75945d0dadebea6 (127.0.98.2:38471)
I20260928 04:54:52.933444 10949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68fa61b0e6a54f298585c3a4083611f1" candidate_uuid: "90bab08e8aad40329a5f2e98aa4b5721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "373e2c6f54c44deab75945d0dadebea6"
I20260928 04:54:52.933476 11083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68fa61b0e6a54f298585c3a4083611f1" candidate_uuid: "90bab08e8aad40329a5f2e98aa4b5721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a7236217074a7a9abc3133282d5d64"
I20260928 04:54:52.933552 11083 raft_consensus.cc:3037] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:52.933540 10949 raft_consensus.cc:3037] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:54:52.934530 10949 raft_consensus.cc:2454] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90bab08e8aad40329a5f2e98aa4b5721 in term 1.
I20260928 04:54:52.934530 11083 raft_consensus.cc:2454] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90bab08e8aad40329a5f2e98aa4b5721 in term 1.
I20260928 04:54:52.934695 10748 leader_election.cc:302] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a7236217074a7a9abc3133282d5d64, 90bab08e8aad40329a5f2e98aa4b5721; no voters:
I20260928 04:54:52.934805 11152 raft_consensus.cc:2781] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:54:52.934959 11152 raft_consensus.cc:686] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [term 1 LEADER]: Becoming Leader. State: Replica: 90bab08e8aad40329a5f2e98aa4b5721, State: Running, Role: LEADER
I20260928 04:54:52.935166 11152 consensus_queue.cc:237] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a7236217074a7a9abc3133282d5d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39463 } } peers { permanent_uuid: "90bab08e8aad40329a5f2e98aa4b5721" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45491 } } peers { permanent_uuid: "373e2c6f54c44deab75945d0dadebea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38471 } }
I20260928 04:54:52.936105 10670 catalog_manager.cc:5495] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90bab08e8aad40329a5f2e98aa4b5721 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "90bab08e8aad40329a5f2e98aa4b5721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a7236217074a7a9abc3133282d5d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90bab08e8aad40329a5f2e98aa4b5721" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "373e2c6f54c44deab75945d0dadebea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38471 } health_report { overall_health: UNKNOWN } } }
I20260928 04:54:52.941504 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.938933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.0.98.1:45491)
0928 04:54:52.938975 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:54:52.938977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:54:52.939012 (+ 35us) server_negotiation.cc:407] Connection header received
0928 04:54:52.939050 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:54:52.939051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:54:52.939070 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:54:52.939130 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:54:52.939232 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.939781 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.940639 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:54:52.940872 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:54:52.940931 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:54:52.941139 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:54:52.941246 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:54:52.941343 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 04:54:52.941424 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 04:54:52.941444 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:54:52.943650 10949 raft_consensus.cc:1261] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6 [term 1 FOLLOWER]: Refusing update from remote peer 90bab08e8aad40329a5f2e98aa4b5721: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:54:52.943650 11083 raft_consensus.cc:1261] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64 [term 1 FOLLOWER]: Refusing update from remote peer 90bab08e8aad40329a5f2e98aa4b5721: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:54:52.944090 11157 consensus_queue.cc:1035] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22a7236217074a7a9abc3133282d5d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:54:52.944324 11152 consensus_queue.cc:1035] T 68fa61b0e6a54f298585c3a4083611f1 P 90bab08e8aad40329a5f2e98aa4b5721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "373e2c6f54c44deab75945d0dadebea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:54:52.949831 10670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:54:52.949975 10670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:54:52.950881 10670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9cc5b3d02a040febf228e0948d96880: Sending DeleteTablet for 3 replicas of tablet 68fa61b0e6a54f298585c3a4083611f1
I20260928 04:54:52.951316 10929 tablet_service.cc:1514] Processing DeleteTablet for tablet 68fa61b0e6a54f298585c3a4083611f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:54:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35898
I20260928 04:54:52.951315 10795 tablet_service.cc:1514] Processing DeleteTablet for tablet 68fa61b0e6a54f298585c3a4083611f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:54:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40828
I20260928 04:54:52.951610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 10732
I20260928 04:54:52.951651 11063 tablet_service.cc:1514] Processing DeleteTablet for tablet 68fa61b0e6a54f298585c3a4083611f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:54:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41240
I20260928 04:54:52.952347 11172 tablet_replica.cc:330] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64: stopping tablet replica
I20260928 04:54:52.952468 11172 raft_consensus.cc:2229] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:54:52.952620 11172 raft_consensus.cc:2258] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:54:52.953377 11172 ts_tablet_manager.cc:1905] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:54:52.953872 11171 tablet_replica.cc:330] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6: stopping tablet replica
I20260928 04:54:52.954023 11171 raft_consensus.cc:2229] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:54:52.954159 11171 raft_consensus.cc:2258] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:54:52.954712 11171 ts_tablet_manager.cc:1905] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:54:52.956216 11172 ts_tablet_manager.cc:1918] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:54:52.956317 11172 log.cc:1196] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/68fa61b0e6a54f298585c3a4083611f1
I20260928 04:54:52.956619 11172 ts_tablet_manager.cc:1939] T 68fa61b0e6a54f298585c3a4083611f1 P 22a7236217074a7a9abc3133282d5d64: Deleting consensus metadata
I20260928 04:54:52.957244 11171 ts_tablet_manager.cc:1918] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:54:52.957396 11171 log.cc:1196] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/68fa61b0e6a54f298585c3a4083611f1
I20260928 04:54:52.957602 10654 catalog_manager.cc:4841] TS 22a7236217074a7a9abc3133282d5d64 (127.0.98.3:39463): tablet 68fa61b0e6a54f298585c3a4083611f1 (table test-table [id=38df6ac4591f4e89b0d2e6a3a15a03bd]) successfully deleted
I20260928 04:54:52.957741 11171 ts_tablet_manager.cc:1939] T 68fa61b0e6a54f298585c3a4083611f1 P 373e2c6f54c44deab75945d0dadebea6: Deleting consensus metadata
I20260928 04:54:52.958396 10657 catalog_manager.cc:4841] TS 373e2c6f54c44deab75945d0dadebea6 (127.0.98.2:38471): tablet 68fa61b0e6a54f298585c3a4083611f1 (table test-table [id=38df6ac4591f4e89b0d2e6a3a15a03bd]) successfully deleted
W20260928 04:54:52.958694 10654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45491 (error 108)
I20260928 04:54:52.958967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 10866
I20260928 04:54:52.959630 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:54:52.959175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45491 (local address 127.0.0.1:40832)
0928 04:54:52.959548 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 04:54:52.959594 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350}
W20260928 04:54:52.959797 10654 catalog_manager.cc:4584] TS 90bab08e8aad40329a5f2e98aa4b5721 (127.0.98.1:45491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 68fa61b0e6a54f298585c3a4083611f1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45491: connect: Connection refused (error 111)
I20260928 04:54:52.965642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 11000
I20260928 04:54:52.970371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 10638
2026-09-28T04:54:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2131 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:54:53 dist-test-slave-m39l krb5kdc[11176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:54:53 dist-test-slave-m39l krb5kdc[11176](info): set up 2 sockets
Sep 28 04:54:53 dist-test-slave-m39l krb5kdc[11176](info): commencing operation
krb5kdc: starting...
W20260928 04:54:54.066875 392 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:54:54 dist-test-slave-m39l krb5kdc[11176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:54:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:54:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 04:54:54.258895 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwNdUqW/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/postgres -l logfile start
2026-09-28 04:54:57.365 UTC [11206] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 04:54:57.365 UTC [11206] LOG: listening on IPv4 address "127.0.98.20", port 56989
2026-09-28 04:54:57.370 UTC [11206] LOG: listening on Unix socket "/tmp/.s.PGSQL.56989"
2026-09-28 04:54:57.396 UTC [11209] LOG: database system was shut down at 2026-09-28 04:54:55 UTC
2026-09-28 04:54:57.400 UTC [11206] LOG: database system is ready to accept connections
I20260928 04:54:58.358816 392 mini_postgres.cc:96] Postgres bound to 56989
2026-09-28 04:54:58.365 UTC [11217] FATAL: database "slave" does not exist
127.0.98.20:56989 - accepting connections
I20260928 04:54:58.366602 392 mini_ranger.cc:161] Starting Ranger...
I20260928 04:54:58.382457 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 04:54:59.021751 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 04:54:59.021843 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ranger-admin
2026-09-28 04:54:59,313 [I] DB FLAVOR :POSTGRES
2026-09-28 04:54:59,313 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 04:54:59,313 [I] Checking connection..
2026-09-28 04:54:59,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 04:54:59,589 [I] Checking connection passed.
2026-09-28 04:54:59,589 [I] --------- Verifying version history table ---------
2026-09-28 04:54:59,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 04:54:59,834 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 04:54:59,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 04:55:00,099 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 04:55:00,099 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 04:55:00,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 04:55:00.355 UTC [11281] WARNING: there is no transaction in progress
2026-09-28 04:55:00,361 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 04:55:00,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 04:55:00,636 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 04:55:00,636 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 04:55:00,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 04:55:00,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 04:55:01,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m39l.c.gcp-upstream.internal','N') ;"
2026-09-28 04:55:01,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 04:55:01,655 [I] Table x_portal_user does not exist in database ranger
2026-09-28 04:55:01,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 04:55:01,896 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 04:55:01,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 04:55:02,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 04:55:02,434 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 04:55:02,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 04:55:03.637 UTC [11389] WARNING: there is no transaction in progress
2026-09-28 04:55:03.656 UTC [11389] WARNING: there is no transaction in progress
2026-09-28 04:55:03.678 UTC [11389] WARNING: there is no transaction in progress
2026-09-28 04:55:03.699 UTC [11389] WARNING: there is no transaction in progress
2026-09-28 04:55:03.719 UTC [11389] WARNING: there is no transaction in progress
2026-09-28 04:55:03.742 UTC [11389] WARNING: there is no transaction in progress
2026-09-28 04:55:03.811 UTC [11389] WARNING: there is no transaction in progress
2026-09-28 04:55:03.821 UTC [11389] WARNING: there is no transaction in progress
2026-09-28 04:55:03.841 UTC [11389] WARNING: there is no transaction in progress
2026-09-28 04:55:03.854 UTC [11389] WARNING: there is no transaction in progress
2026-09-28 04:55:03.866 UTC [11389] WARNING: there is no transaction in progress
2026-09-28 04:55:03.883 UTC [11389] WARNING: there is no transaction in progress
2026-09-28 04:55:03.895 UTC [11389] WARNING: there is no transaction in progress
2026-09-28 04:55:03.903 UTC [11389] WARNING: there is no transaction in progress
2026-09-28 04:55:03.913 UTC [11389] WARNING: there is no transaction in progress
2026-09-28 04:55:03.922 UTC [11389] WARNING: there is no transaction in progress
2026-09-28 04:55:04,776 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 04:55:04,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 04:55:05,018 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 04:55:05,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 04:55:05,276 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 04:55:05,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 04:55:05,538 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 04:55:05,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 04:55:05,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 04:55:06,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m39l.c.gcp-upstream.internal';"
2026-09-28 04:55:06,334 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 04:55:06,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 04:55:06,600 [I] DB_PATCHES have already been applied
I20260928 04:55:06.607605 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwNdUqW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 04:55:06.607679 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 04:55:06.611465 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:56671
Sep 28, 2026 4:55:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:55:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:55:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 4:55:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 4:55:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 4:55:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 4:55:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 04:55:07.743964 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.377s user 0.007s sys 0.003s
Sep 28, 2026 4:55:28 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 4:55:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 4:55:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 4:55:28 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 4:55:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 4:55:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 4:55:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 4:55:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 4:55:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 4:55:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 4:55:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 4:55:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 4:55:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 4:55:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 4:55:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 4:55:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 4:55:28 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 4:55:29 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 04:55:30.068670 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 04:55:30 dist-test-slave-m39l krb5kdc[11176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571330, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 04:55:30.153843 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 04:55:30.163197 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 04:55:31.194257 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 04:55:31.194345 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ranger-kms
I20260928 04:55:31.194368 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:56989
2026-09-28 04:55:31,377 [I] DB FLAVOR :POSTGRES
2026-09-28 04:55:31,377 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 04:55:31,378 [I] Checking connection
2026-09-28 04:55:31,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 04:55:31,632 [I] connection success
2026-09-28 04:55:31,633 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 04:55:31,633 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 04:55:31,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 04:55:31,890 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 04:55:31,890 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 04:55:31,891 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 04:55:31,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56989/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 04:55:32,175 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 04:55:32.712090 392 mini_ranger_kms.cc:310] Created kms service
I20260928 04:55:32.938093 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 04:55:32.985117 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 04:55:32 dist-test-slave-m39l krb5kdc[11176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571330, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:55:33.469825 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 04:55:33.469950 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 04:55:33.469975 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 04:55:33.473659 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11570
I20260928 04:55:33.473817 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:37835
Sep 28, 2026 4:55:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:55:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:55:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 4:55:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 4:55:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 4:55:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 04:55:35.774868 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.621s user 0.003s sys 0.162s
I20260928 04:55:35.774957 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 04:55:35.775019 392 mini_ranger_kms.cc:401] 127.0.98.20:37835/kms/v1/keys
Sep 28 04:55:35 dist-test-slave-m39l krb5kdc[11176](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790571335, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 04:55:36 dist-test-slave-m39l krb5kdc[11176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571335, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 04:55:36 dist-test-slave-m39l krb5kdc[11176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571335, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 04:55:39 dist-test-slave-m39l krb5kdc[11176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:55:39.143774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40235
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40235
--ranger_config_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37835/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:55:39.243042 11620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:55:39.243270 11620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:55:39.243294 11620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:55:39.244588 11620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:55:39.244632 11620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:55:39.244645 11620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:55:39.244657 11620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:55:39.244669 11620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:55:39.246326 11620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:37835/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40235
--ranger_config_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:55:39.246598 11620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:55:39.246845 11620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:55:39.252923 11620 server_base.cc:1034] running on GCE node
W20260928 04:55:39.252871 11626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:55:39.252995 11628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:55:39.252888 11625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:55:39.253460 11620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:55:39.253763 11620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:55:39.254957 11620 hybrid_clock.cc:648] HybridClock initialized: now 1790571339254936 us; error 44 us; skew 500 ppm
Sep 28 04:55:39 dist-test-slave-m39l krb5kdc[11176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:55:39.263795 11620 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:55:39.264369 11620 webserver.cc:460] Webserver started at http://127.0.98.62:33131/ using document root <none> and password file <none>
I20260928 04:55:39.264616 11620 fs_manager.cc:359] Metadata directory not provided
I20260928 04:55:39.264678 11620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:55:39.264842 11620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:55:39 dist-test-slave-m39l krb5kdc[11176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:55:39.325577 11620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "350c92cbf80e44db8f0b679c4b103a92"
format_stamp: "Formatted at 2026-09-28 04:55:39 on dist-test-slave-m39l"
server_key: "5e156043f47cd89d360b319e52ca7832"
server_key_iv: "3cf5561b6be55be92ae7bac299580ac0"
server_key_version: "kuduclusterkey@0"
I20260928 04:55:39.326023 11620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "350c92cbf80e44db8f0b679c4b103a92"
format_stamp: "Formatted at 2026-09-28 04:55:39 on dist-test-slave-m39l"
server_key: "5e156043f47cd89d360b319e52ca7832"
server_key_iv: "3cf5561b6be55be92ae7bac299580ac0"
server_key_version: "kuduclusterkey@0"
I20260928 04:55:39.327631 11620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 04:55:39.342881 11635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:55:39.343067 11620 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:55:39.343168 11620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "350c92cbf80e44db8f0b679c4b103a92"
format_stamp: "Formatted at 2026-09-28 04:55:39 on dist-test-slave-m39l"
server_key: "5e156043f47cd89d360b319e52ca7832"
server_key_iv: "3cf5561b6be55be92ae7bac299580ac0"
server_key_version: "kuduclusterkey@0"
I20260928 04:55:39.343277 11620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:55:39.364316 11620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:55:39.366632 11620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:55:39.366798 11620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:55:39.372283 11620 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40235
I20260928 04:55:39.372301 11687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40235 every 8 connection(s)
I20260928 04:55:39.372731 11620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:55:39.373457 11688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:55:39.375958 11688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92: Bootstrap starting.
I20260928 04:55:39.377032 11688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92: Neither blocks nor log segments found. Creating new log.
I20260928 04:55:39.377326 11688 log.cc:824] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92: Log is configured to *not* fsync() on all Append() calls
I20260928 04:55:39.378095 11688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92: No bootstrap required, opened a new log
I20260928 04:55:39.380225 11688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350c92cbf80e44db8f0b679c4b103a92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40235 } }
I20260928 04:55:39.380371 11688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:55:39.380394 11688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 350c92cbf80e44db8f0b679c4b103a92, State: Initialized, Role: FOLLOWER
I20260928 04:55:39.380551 11688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350c92cbf80e44db8f0b679c4b103a92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40235 } }
I20260928 04:55:39.380633 11688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:55:39.380685 11688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:55:39.380735 11688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:55:39.381146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 11620
I20260928 04:55:39.381240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:55:39.381438 392 external_mini_cluster.cc:1428] Setting key 743f4a69de56f2b71c211bb478e05218
I20260928 04:55:39.381561 11688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350c92cbf80e44db8f0b679c4b103a92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40235 } }
I20260928 04:55:39.381667 11688 leader_election.cc:302] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 350c92cbf80e44db8f0b679c4b103a92; no voters:
I20260928 04:55:39.381880 11688 leader_election.cc:288] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:55:39.382130 11694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:55:39.382246 11688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:55:39.382270 11694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92 [term 1 LEADER]: Becoming Leader. State: Replica: 350c92cbf80e44db8f0b679c4b103a92, State: Running, Role: LEADER
I20260928 04:55:39.382350 11694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350c92cbf80e44db8f0b679c4b103a92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40235 } }
I20260928 04:55:39.382705 11694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "350c92cbf80e44db8f0b679c4b103a92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350c92cbf80e44db8f0b679c4b103a92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40235 } } }
I20260928 04:55:39.383443 11694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92 [sys.catalog]: This master's current role is: LEADER
I20260928 04:55:39.383654 11694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 350c92cbf80e44db8f0b679c4b103a92. Latest consensus state: current_term: 1 leader_uuid: "350c92cbf80e44db8f0b679c4b103a92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350c92cbf80e44db8f0b679c4b103a92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40235 } } }
I20260928 04:55:39.383735 11702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:55:39.383767 11694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92 [sys.catalog]: This master's current role is: LEADER
I20260928 04:55:39.384413 11702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:55:39.384723 11688 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 04:55:39.390695 11702 catalog_manager.cc:1347] Generated new cluster ID: ddd7bb6e8e474bf4b645d19e5cb408bd
I20260928 04:55:39.390869 11702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:55:39 dist-test-slave-m39l krb5kdc[11176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:55:39.405632 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:39.381986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.0.98.62:40235)
0928 04:55:39.383393 (+ 1407us) server_negotiation.cc:206] Beginning negotiation
0928 04:55:39.383403 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:55:39.390798 (+ 7395us) server_negotiation.cc:407] Connection header received
0928 04:55:39.391140 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:55:39.391149 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:55:39.391240 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:55:39.391389 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:55:39.394759 (+ 3370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:39.396379 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:39.397478 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:39.398172 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:39.398274 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:39.401321 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:55:39.401326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:55:39.401326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:55:39.401342 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:55:39.403642 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:55:39.404066 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:55:39.404069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:55:39.404070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:55:39.404182 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:55:39.404367 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:55:39.404368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:55:39.404368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:55:39.404736 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:55:39.404869 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 04:55:39.405132 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 04:55:39.405334 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892,"thread_start_us":73,"threads_started":1}
I20260928 04:55:39.425463 11702 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:55:39.426633 11702 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:55:39.438241 11702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92: Generated new TSK 0
I20260928 04:55:39.438848 11702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:55:40.202601 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40235
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:55:40.301796 11733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:55:40.302066 11733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:55:40.302111 11733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:55:40.303436 11733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:55:40.303512 11733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:55:40.303562 11733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:55:40.305148 11733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:37835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:55:40.305444 11733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:55:40.305706 11733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:55:40.312121 11738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:55:40.312121 11739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:55:40.312412 11733 server_base.cc:1034] running on GCE node
W20260928 04:55:40.312121 11741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:55:40.312806 11733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:55:40.313104 11733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:55:40.314342 11733 hybrid_clock.cc:648] HybridClock initialized: now 1790571340314335 us; error 36 us; skew 500 ppm
Sep 28 04:55:40 dist-test-slave-m39l krb5kdc[11176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:55:40.317878 11733 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:55:40.318567 11733 webserver.cc:460] Webserver started at http://127.0.98.1:45665/ using document root <none> and password file <none>
I20260928 04:55:40.318884 11733 fs_manager.cc:359] Metadata directory not provided
I20260928 04:55:40.318984 11733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:55:40.319149 11733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:55:40 dist-test-slave-m39l krb5kdc[11176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:55:40.336865 11733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "fdd99ac38b684d6795804cbbecff1bf8"
format_stamp: "Formatted at 2026-09-28 04:55:40 on dist-test-slave-m39l"
server_key: "e3171522009e4c116212317820848a18"
server_key_iv: "580b52490e80938c541173a5940dcbda"
server_key_version: "kuduclusterkey@0"
I20260928 04:55:40.337406 11733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fdd99ac38b684d6795804cbbecff1bf8"
format_stamp: "Formatted at 2026-09-28 04:55:40 on dist-test-slave-m39l"
server_key: "e3171522009e4c116212317820848a18"
server_key_iv: "580b52490e80938c541173a5940dcbda"
server_key_version: "kuduclusterkey@0"
I20260928 04:55:40.339358 11733 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 04:55:40.355767 11748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:55:40.355976 11733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 04:55:40.356096 11733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "fdd99ac38b684d6795804cbbecff1bf8"
format_stamp: "Formatted at 2026-09-28 04:55:40 on dist-test-slave-m39l"
server_key: "e3171522009e4c116212317820848a18"
server_key_iv: "580b52490e80938c541173a5940dcbda"
server_key_version: "kuduclusterkey@0"
I20260928 04:55:40.356225 11733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:55:40.377368 11733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:55:40.379997 11733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:55:40.380162 11733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:55:40.380403 11733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:55:40.380759 11733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:55:40.380808 11733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:55:40.380839 11733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:55:40.380857 11733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:55:40.388701 11733 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42465
I20260928 04:55:40.389154 11733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:55:40.390305 11863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42465 every 8 connection(s)
I20260928 04:55:40.391259 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 11733
I20260928 04:55:40.391361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:55:40.391589 392 external_mini_cluster.cc:1428] Setting key c93d3f082ab4663b48381b520aaea032
Sep 28 04:55:40 dist-test-slave-m39l krb5kdc[11176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:55:40.406936 11867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:40.390999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40235 (local address 127.0.98.1:52783)
0928 04:55:40.392046 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0928 04:55:40.392098 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 04:55:40.392815 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:55:40.394108 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:55:40.394113 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:55:40.394363 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:55:40.394686 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:55:40.394692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.396527 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.396528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:55:40.399176 (+ 2648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:55:40.399179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.399754 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.399755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:55:40.399911 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:40.400582 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:55:40.400593 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:55:40.402339 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:55:40.405027 (+ 2688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:55:40.405035 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:55:40.405038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:55:40.405377 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:55:40.405622 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:55:40.405623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:55:40.405623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:55:40.405777 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:55:40.406122 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:55:40.406128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:55:40.406326 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 04:55:40.406454 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 04:55:40.406666 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"thread_start_us":105,"threads_started":1}
I20260928 04:55:40.407310 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:40.391012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52783 (local address 127.0.98.62:40235)
0928 04:55:40.393010 (+ 1998us) server_negotiation.cc:206] Beginning negotiation
0928 04:55:40.393012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:55:40.393033 (+ 21us) server_negotiation.cc:407] Connection header received
0928 04:55:40.393058 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:55:40.393059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:55:40.393072 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:55:40.393955 (+ 883us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:55:40.394798 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.396407 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.399287 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.399618 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.399683 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:40.402467 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:55:40.402469 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:55:40.402469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:55:40.402478 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:55:40.404923 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:55:40.405469 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:55:40.405470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:55:40.405471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:55:40.405536 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:55:40.405867 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:55:40.405868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:55:40.405869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:55:40.406023 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:55:40.406090 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 04:55:40.407110 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0928 04:55:40.407211 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1965,"thread_start_us":83,"threads_started":1}
I20260928 04:55:40.407737 11864 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40235
I20260928 04:55:40.407888 11864 heartbeater.cc:464] Registering TS with master...
I20260928 04:55:40.408134 11864 heartbeater.cc:511] Master 127.0.98.62:40235 requested a full tablet report, sending...
I20260928 04:55:40.408725 11652 ts_manager.cc:194] Registered new tserver with Master: fdd99ac38b684d6795804cbbecff1bf8 (127.0.98.1:42465)
I20260928 04:55:40.409662 11652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52783
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:55:40.434780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40235
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:55:40.547797 11870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:55:40.548085 11870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:55:40.548131 11870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:55:40.549458 11870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:55:40.549541 11870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:55:40.549589 11870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:55:40.551393 11870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:37835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:55:40.551710 11870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:55:40.552011 11870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:55:40.557361 11878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:55:40.557345 11880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:55:40.557610 11877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:55:40.557999 11870 server_base.cc:1034] running on GCE node
I20260928 04:55:40.558254 11870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:55:40.558568 11870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:55:40.559818 11870 hybrid_clock.cc:648] HybridClock initialized: now 1790571340559762 us; error 30 us; skew 500 ppm
Sep 28 04:55:40 dist-test-slave-m39l krb5kdc[11176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:55:40.562899 11870 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:55:40.563385 11870 webserver.cc:460] Webserver started at http://127.0.98.2:34665/ using document root <none> and password file <none>
I20260928 04:55:40.563653 11870 fs_manager.cc:359] Metadata directory not provided
I20260928 04:55:40.563726 11870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:55:40.563892 11870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:55:40 dist-test-slave-m39l krb5kdc[11176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:55:40.578853 11870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "9ecd038069ac49c5872cd33a75ca26f0"
format_stamp: "Formatted at 2026-09-28 04:55:40 on dist-test-slave-m39l"
server_key: "239d1f05c77cfe27fd02240e7e478820"
server_key_iv: "71e788cf9002bafab4a54a1a1101114e"
server_key_version: "kuduclusterkey@0"
I20260928 04:55:40.579448 11870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9ecd038069ac49c5872cd33a75ca26f0"
format_stamp: "Formatted at 2026-09-28 04:55:40 on dist-test-slave-m39l"
server_key: "239d1f05c77cfe27fd02240e7e478820"
server_key_iv: "71e788cf9002bafab4a54a1a1101114e"
server_key_version: "kuduclusterkey@0"
Sep 28 04:55:40 dist-test-slave-m39l krb5kdc[11176](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790571340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:55:40.590679 11870 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.001s sys 0.000s
I20260928 04:55:40.608440 11887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:55:40.608664 11870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260928 04:55:40.608755 11870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "9ecd038069ac49c5872cd33a75ca26f0"
format_stamp: "Formatted at 2026-09-28 04:55:40 on dist-test-slave-m39l"
server_key: "239d1f05c77cfe27fd02240e7e478820"
server_key_iv: "71e788cf9002bafab4a54a1a1101114e"
server_key_version: "kuduclusterkey@0"
I20260928 04:55:40.608866 11870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:55:40.636847 11870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:55:40.639322 11870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:55:40.639474 11870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:55:40.639698 11870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:55:40.640089 11870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:55:40.640138 11870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:55:40.640182 11870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:55:40.640205 11870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:55:40.647408 11870 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34311
I20260928 04:55:40.647804 11870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:55:40.651672 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 11870
I20260928 04:55:40.651790 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:55:40.651998 392 external_mini_cluster.cc:1428] Setting key 09b7352fed56d40dd7280e24546da20a
I20260928 04:55:40.654363 12000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34311 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 28 04:55:40 dist-test-slave-m39l krb5kdc[11176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:55:40.673848 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:40.655095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39461 (local address 127.0.98.62:40235)
0928 04:55:40.655139 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:55:40.655141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:55:40.655399 (+ 258us) server_negotiation.cc:407] Connection header received
0928 04:55:40.656210 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:55:40.656211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:55:40.656224 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:55:40.656254 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:55:40.657161 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.658016 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.667336 (+ 9320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.667694 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.667748 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:40.670317 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:55:40.670318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:55:40.670319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:55:40.670335 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:55:40.672129 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:55:40.672633 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:55:40.672634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:55:40.672635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:55:40.672693 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:55:40.672966 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:55:40.672966 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:55:40.672967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:55:40.673131 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:55:40.673179 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 04:55:40.673612 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 04:55:40.673719 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:55:40.674665 12004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:40.655081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40235 (local address 127.0.98.2:39461)
0928 04:55:40.655270 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 04:55:40.655322 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 04:55:40.656085 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:55:40.656444 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:55:40.656449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:55:40.656733 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:55:40.657033 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:55:40.657051 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.663770 (+ 6719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.663771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:55:40.667202 (+ 3431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:55:40.667208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.667854 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.667855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:55:40.668015 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:40.668696 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:55:40.668707 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:55:40.670192 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:55:40.672221 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:55:40.672228 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:55:40.672231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:55:40.672543 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:55:40.672760 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:55:40.672760 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:55:40.672761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:55:40.672899 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:55:40.673227 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:55:40.673233 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:55:40.673436 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 04:55:40.674240 (+ 804us) client_negotiation.cc:240] Negotiation successful
0928 04:55:40.674374 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260928 04:55:40.678032 12001 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40235
I20260928 04:55:40.678148 12001 heartbeater.cc:464] Registering TS with master...
I20260928 04:55:40.678360 12001 heartbeater.cc:511] Master 127.0.98.62:40235 requested a full tablet report, sending...
I20260928 04:55:40.678808 11652 ts_manager.cc:194] Registered new tserver with Master: 9ecd038069ac49c5872cd33a75ca26f0 (127.0.98.2:34311)
I20260928 04:55:40.679328 11652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:55:40.689162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40235
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:55:40.817672 12007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:55:40.817987 12007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:55:40.818034 12007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:55:40.819994 12007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:55:40.820065 12007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:55:40.820111 12007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:55:40.822481 12007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:37835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:55:40.822814 12007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:55:40.823143 12007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:55:40.830228 12012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:55:40.830228 12013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:55:40.830232 12015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:55:40.831029 12007 server_base.cc:1034] running on GCE node
I20260928 04:55:40.831252 12007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:55:40.831534 12007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:55:40.832739 12007 hybrid_clock.cc:648] HybridClock initialized: now 1790571340832711 us; error 41 us; skew 500 ppm
Sep 28 04:55:40 dist-test-slave-m39l krb5kdc[11176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:55:40.836617 12007 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:55:40.837301 12007 webserver.cc:460] Webserver started at http://127.0.98.3:40135/ using document root <none> and password file <none>
I20260928 04:55:40.838085 12007 fs_manager.cc:359] Metadata directory not provided
I20260928 04:55:40.838246 12007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:55:40.838469 12007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:55:40 dist-test-slave-m39l krb5kdc[11176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:55:40.864262 12007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "93cfc647c10a4315a688fdf059f6514a"
format_stamp: "Formatted at 2026-09-28 04:55:40 on dist-test-slave-m39l"
server_key: "8badaecae7e4f9c0a2e11de711ebbb43"
server_key_iv: "1c4b58c6e8fb3309d35f084a33b6c432"
server_key_version: "kuduclusterkey@0"
I20260928 04:55:40.864859 12007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "93cfc647c10a4315a688fdf059f6514a"
format_stamp: "Formatted at 2026-09-28 04:55:40 on dist-test-slave-m39l"
server_key: "8badaecae7e4f9c0a2e11de711ebbb43"
server_key_iv: "1c4b58c6e8fb3309d35f084a33b6c432"
server_key_version: "kuduclusterkey@0"
I20260928 04:55:40.869108 12007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260928 04:55:40.888506 12022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:55:40.888763 12007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260928 04:55:40.888854 12007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "93cfc647c10a4315a688fdf059f6514a"
format_stamp: "Formatted at 2026-09-28 04:55:40 on dist-test-slave-m39l"
server_key: "8badaecae7e4f9c0a2e11de711ebbb43"
server_key_iv: "1c4b58c6e8fb3309d35f084a33b6c432"
server_key_version: "kuduclusterkey@0"
I20260928 04:55:40.888952 12007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:55:40.904259 12007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:55:40.906903 12007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:55:40.907099 12007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:55:40.907356 12007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:55:40.907768 12007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:55:40.907828 12007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:55:40.907869 12007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:55:40.907891 12007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:55:40.915879 12007 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37909
I20260928 04:55:40.916033 12135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37909 every 8 connection(s)
I20260928 04:55:40.916301 12007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:55:40.917594 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 12007
I20260928 04:55:40.917695 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:55:40.917891 392 external_mini_cluster.cc:1428] Setting key a18784e0cdced3ea88cb37cd3bc19169
Sep 28 04:55:40 dist-test-slave-m39l krb5kdc[11176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:55:40.926756 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:40.917108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60463 (local address 127.0.98.62:40235)
0928 04:55:40.917158 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:55:40.917161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:55:40.917391 (+ 230us) server_negotiation.cc:407] Connection header received
0928 04:55:40.918154 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:55:40.918156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:55:40.918171 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:55:40.918202 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:55:40.919081 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.919651 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.921167 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.921365 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.921460 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:40.923723 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:55:40.923724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:55:40.923725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:55:40.923744 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:55:40.925233 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:55:40.925684 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:55:40.925686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:55:40.925686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:55:40.925748 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:55:40.926023 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:55:40.926024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:55:40.926024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:55:40.926148 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:55:40.926228 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:55:40.926570 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 04:55:40.926659 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:55:40.926959 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:40.917113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40235 (local address 127.0.98.3:60463)
0928 04:55:40.917267 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 04:55:40.917310 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:55:40.918020 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:55:40.918400 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:55:40.918404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:55:40.918689 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:55:40.918979 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:55:40.918984 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.919744 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.919745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:55:40.921052 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:55:40.921055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.921469 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.921470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:55:40.921615 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:40.922223 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:55:40.922232 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:55:40.923612 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:55:40.925341 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:55:40.925349 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:55:40.925352 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:55:40.925602 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:55:40.925819 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:55:40.925820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:55:40.925820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:55:40.925951 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:55:40.926219 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:55:40.926225 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:55:40.926402 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 04:55:40.926546 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 04:55:40.926713 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260928 04:55:40.927482 12136 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40235
I20260928 04:55:40.927594 12136 heartbeater.cc:464] Registering TS with master...
I20260928 04:55:40.927811 12136 heartbeater.cc:511] Master 127.0.98.62:40235 requested a full tablet report, sending...
I20260928 04:55:40.928236 11652 ts_manager.cc:194] Registered new tserver with Master: 93cfc647c10a4315a688fdf059f6514a (127.0.98.3:37909)
I20260928 04:55:40.928624 11652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60463
I20260928 04:55:40.929502 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:55:40.940711 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:40.931110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40866 (local address 127.0.98.62:40235)
0928 04:55:40.931148 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 04:55:40.931150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:55:40.931717 (+ 567us) server_negotiation.cc:407] Connection header received
0928 04:55:40.931750 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:55:40.931752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:55:40.931765 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:55:40.931840 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:55:40.932544 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.933345 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.934125 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.934436 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.934539 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:40.936170 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:55:40.936171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:55:40.936171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:55:40.936180 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:55:40.937890 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:55:40.938277 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:55:40.938278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:55:40.938279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:55:40.938352 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:55:40.938576 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:55:40.938577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:55:40.938577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:55:40.938718 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:55:40.938819 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 04:55:40.938931 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 04:55:40.940616 (+ 1685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 04:55:40.942752 11652 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:55:40.945041 11652 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:55:40.955986 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:40.951202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42465 (local address 127.0.0.1:48792)
0928 04:55:40.951478 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 04:55:40.951523 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:55:40.951607 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:55:40.951984 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:55:40.951987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:55:40.951990 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:55:40.952068 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:55:40.952070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.953387 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.953388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:55:40.954733 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:55:40.954736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.955681 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.955682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:55:40.955768 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:40.955786 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 04:55:40.955832 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 04:55:40.955888 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":70,"threads_started":1}
I20260928 04:55:40.955991 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:40.951255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34311 (local address 127.0.0.1:39862)
0928 04:55:40.951722 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 04:55:40.951730 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:55:40.951767 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:55:40.952547 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:55:40.952548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:55:40.952550 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:55:40.952591 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:55:40.952593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.953835 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.953836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:55:40.954916 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:55:40.954918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.955799 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.955800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:55:40.955852 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:40.955855 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:55:40.955881 (+ 26us) client_negotiation.cc:240] Negotiation successful
0928 04:55:40.955919 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":95,"threads_started":1}
I20260928 04:55:40.956284 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:40.951734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39862 (local address 127.0.98.2:34311)
0928 04:55:40.951904 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 04:55:40.951908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:55:40.951917 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:55:40.952402 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:55:40.952406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:55:40.952425 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:55:40.952654 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:55:40.952718 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.953738 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.955018 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.955717 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.956029 (+ 312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:40.956118 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 04:55:40.956145 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 04:55:40.956188 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260928 04:55:40.956496 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:40.951458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.0.98.1:42465)
0928 04:55:40.951803 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0928 04:55:40.951809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:55:40.951819 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:55:40.951859 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:55:40.951861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:55:40.951877 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:55:40.952150 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:55:40.952211 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.953291 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.954827 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.955583 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.956318 (+ 735us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:40.956387 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:55:40.956415 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 04:55:40.956445 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":48,"threads_started":1}
I20260928 04:55:40.957113 11791 tablet_service.cc:1467] Processing CreateTablet for tablet c1cdef4ad8334bf7bc731ae7743be10b (DEFAULT_TABLE table=test-table [id=be721f4d8b95453587d2e93c1261276e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:55:40.957085 11923 tablet_service.cc:1467] Processing CreateTablet for tablet c1cdef4ad8334bf7bc731ae7743be10b (DEFAULT_TABLE table=test-table [id=be721f4d8b95453587d2e93c1261276e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:55:40.957448 11791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1cdef4ad8334bf7bc731ae7743be10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:55:40.957448 11923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1cdef4ad8334bf7bc731ae7743be10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:55:40.961161 12155 tablet_bootstrap.cc:492] T c1cdef4ad8334bf7bc731ae7743be10b P fdd99ac38b684d6795804cbbecff1bf8: Bootstrap starting.
I20260928 04:55:40.961858 12155 tablet_bootstrap.cc:654] T c1cdef4ad8334bf7bc731ae7743be10b P fdd99ac38b684d6795804cbbecff1bf8: Neither blocks nor log segments found. Creating new log.
I20260928 04:55:40.962142 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:40.951202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37909 (local address 127.0.0.1:55120)
0928 04:55:40.951638 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 04:55:40.951647 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:55:40.951687 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:55:40.952435 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:55:40.952436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:55:40.952439 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:55:40.952484 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:55:40.952486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.953959 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.953960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:55:40.955130 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:55:40.955132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.961817 (+ 6685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.961819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:55:40.961907 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:40.961973 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 04:55:40.962009 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 04:55:40.962061 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":4,"thread_start_us":66,"threads_started":1}
I20260928 04:55:40.962659 12155 log.cc:824] T c1cdef4ad8334bf7bc731ae7743be10b P fdd99ac38b684d6795804cbbecff1bf8: Log is configured to *not* fsync() on all Append() calls
I20260928 04:55:40.962847 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:40.951522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.0.98.3:37909)
0928 04:55:40.951930 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0928 04:55:40.951936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:55:40.951949 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:55:40.951993 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:55:40.951995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:55:40.952012 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:55:40.952570 (+ 558us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:55:40.952637 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.953861 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.955252 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.956017 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.962477 (+ 6460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:40.962618 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 04:55:40.962649 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 04:55:40.962698 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":115,"threads_started":1}
I20260928 04:55:40.963429 12155 tablet_bootstrap.cc:492] T c1cdef4ad8334bf7bc731ae7743be10b P fdd99ac38b684d6795804cbbecff1bf8: No bootstrap required, opened a new log
I20260928 04:55:40.963496 12155 ts_tablet_manager.cc:1397] T c1cdef4ad8334bf7bc731ae7743be10b P fdd99ac38b684d6795804cbbecff1bf8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:55:40.963613 12070 tablet_service.cc:1467] Processing CreateTablet for tablet c1cdef4ad8334bf7bc731ae7743be10b (DEFAULT_TABLE table=test-table [id=be721f4d8b95453587d2e93c1261276e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:55:40.963943 12070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1cdef4ad8334bf7bc731ae7743be10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:55:40.966032 12155 raft_consensus.cc:348] T c1cdef4ad8334bf7bc731ae7743be10b P fdd99ac38b684d6795804cbbecff1bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cfc647c10a4315a688fdf059f6514a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } } peers { permanent_uuid: "9ecd038069ac49c5872cd33a75ca26f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34311 } } peers { permanent_uuid: "fdd99ac38b684d6795804cbbecff1bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42465 } }
I20260928 04:55:40.966193 12155 raft_consensus.cc:374] T c1cdef4ad8334bf7bc731ae7743be10b P fdd99ac38b684d6795804cbbecff1bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:55:40.966225 12155 raft_consensus.cc:729] T c1cdef4ad8334bf7bc731ae7743be10b P fdd99ac38b684d6795804cbbecff1bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd99ac38b684d6795804cbbecff1bf8, State: Initialized, Role: FOLLOWER
I20260928 04:55:40.966363 12155 consensus_queue.cc:260] T c1cdef4ad8334bf7bc731ae7743be10b P fdd99ac38b684d6795804cbbecff1bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cfc647c10a4315a688fdf059f6514a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } } peers { permanent_uuid: "9ecd038069ac49c5872cd33a75ca26f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34311 } } peers { permanent_uuid: "fdd99ac38b684d6795804cbbecff1bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42465 } }
I20260928 04:55:40.966642 12155 ts_tablet_manager.cc:1428] T c1cdef4ad8334bf7bc731ae7743be10b P fdd99ac38b684d6795804cbbecff1bf8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:55:40.966753 11864 heartbeater.cc:503] Master 127.0.98.62:40235 was elected leader, sending a full tablet report...
I20260928 04:55:40.968223 12156 tablet_bootstrap.cc:492] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0: Bootstrap starting.
I20260928 04:55:40.969264 12156 tablet_bootstrap.cc:654] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0: Neither blocks nor log segments found. Creating new log.
I20260928 04:55:40.969571 12156 log.cc:824] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0: Log is configured to *not* fsync() on all Append() calls
I20260928 04:55:40.969911 12159 tablet_bootstrap.cc:492] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a: Bootstrap starting.
I20260928 04:55:40.971239 12159 tablet_bootstrap.cc:654] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a: Neither blocks nor log segments found. Creating new log.
I20260928 04:55:40.971529 12159 log.cc:824] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a: Log is configured to *not* fsync() on all Append() calls
I20260928 04:55:40.972353 12159 tablet_bootstrap.cc:492] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a: No bootstrap required, opened a new log
I20260928 04:55:40.972425 12159 ts_tablet_manager.cc:1397] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 04:55:40.975327 12159 raft_consensus.cc:348] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cfc647c10a4315a688fdf059f6514a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } } peers { permanent_uuid: "9ecd038069ac49c5872cd33a75ca26f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34311 } } peers { permanent_uuid: "fdd99ac38b684d6795804cbbecff1bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42465 } }
I20260928 04:55:40.975520 12159 raft_consensus.cc:374] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:55:40.975577 12159 raft_consensus.cc:729] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93cfc647c10a4315a688fdf059f6514a, State: Initialized, Role: FOLLOWER
I20260928 04:55:40.975732 12159 consensus_queue.cc:260] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cfc647c10a4315a688fdf059f6514a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } } peers { permanent_uuid: "9ecd038069ac49c5872cd33a75ca26f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34311 } } peers { permanent_uuid: "fdd99ac38b684d6795804cbbecff1bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42465 } }
I20260928 04:55:40.976046 12159 ts_tablet_manager.cc:1428] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 04:55:40.976178 12136 heartbeater.cc:503] Master 127.0.98.62:40235 was elected leader, sending a full tablet report...
I20260928 04:55:40.976586 12156 tablet_bootstrap.cc:492] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0: No bootstrap required, opened a new log
I20260928 04:55:40.976672 12156 ts_tablet_manager.cc:1397] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260928 04:55:40.979079 12156 raft_consensus.cc:348] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cfc647c10a4315a688fdf059f6514a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } } peers { permanent_uuid: "9ecd038069ac49c5872cd33a75ca26f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34311 } } peers { permanent_uuid: "fdd99ac38b684d6795804cbbecff1bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42465 } }
I20260928 04:55:40.979259 12156 raft_consensus.cc:374] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:55:40.979300 12156 raft_consensus.cc:729] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd038069ac49c5872cd33a75ca26f0, State: Initialized, Role: FOLLOWER
I20260928 04:55:40.979461 12156 consensus_queue.cc:260] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cfc647c10a4315a688fdf059f6514a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } } peers { permanent_uuid: "9ecd038069ac49c5872cd33a75ca26f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34311 } } peers { permanent_uuid: "fdd99ac38b684d6795804cbbecff1bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42465 } }
I20260928 04:55:40.979740 12156 ts_tablet_manager.cc:1428] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 04:55:40.979862 12001 heartbeater.cc:503] Master 127.0.98.62:40235 was elected leader, sending a full tablet report...
I20260928 04:55:40.980845 12163 raft_consensus.cc:482] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:55:40.981149 12163 raft_consensus.cc:504] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cfc647c10a4315a688fdf059f6514a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } } peers { permanent_uuid: "9ecd038069ac49c5872cd33a75ca26f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34311 } } peers { permanent_uuid: "fdd99ac38b684d6795804cbbecff1bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42465 } }
I20260928 04:55:40.981917 12163 leader_election.cc:288] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93cfc647c10a4315a688fdf059f6514a (127.0.98.3:37909), fdd99ac38b684d6795804cbbecff1bf8 (127.0.98.1:42465)
I20260928 04:55:40.985440 12004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:40.981622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42465 (local address 127.0.98.2:58239)
0928 04:55:40.981717 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 04:55:40.981729 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:55:40.981766 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:55:40.982022 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:55:40.982023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:55:40.982025 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:55:40.982191 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:55:40.982195 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.983205 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.983206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:55:40.984389 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:55:40.984392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.985197 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.985198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:55:40.985270 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:40.985273 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:55:40.985321 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 04:55:40.985364 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 04:55:40.985821 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:40.981848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58239 (local address 127.0.98.1:42465)
0928 04:55:40.981905 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:55:40.981908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:55:40.981917 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:55:40.981945 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:55:40.981946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:55:40.981958 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:55:40.982020 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:55:40.982279 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.983095 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.984507 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.985083 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.985153 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:40.985165 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 04:55:40.985720 (+ 555us) server_negotiation.cc:299] Negotiation successful
0928 04:55:40.985759 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:55:40.986107 11818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1cdef4ad8334bf7bc731ae7743be10b" candidate_uuid: "9ecd038069ac49c5872cd33a75ca26f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd99ac38b684d6795804cbbecff1bf8" is_pre_election: true
I20260928 04:55:40.986215 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:40.981805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44897 (local address 127.0.98.3:37909)
0928 04:55:40.981897 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 04:55:40.981899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:55:40.982215 (+ 316us) server_negotiation.cc:407] Connection header received
0928 04:55:40.982304 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:55:40.982305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:55:40.982315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:55:40.982344 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:55:40.982578 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.983305 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.985048 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.985636 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.985682 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:40.985690 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:55:40.986104 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 04:55:40.986151 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 04:55:40.986284 11818 raft_consensus.cc:2454] T c1cdef4ad8334bf7bc731ae7743be10b P fdd99ac38b684d6795804cbbecff1bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd038069ac49c5872cd33a75ca26f0 in term 0.
I20260928 04:55:40.986423 12165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:40.981639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37909 (local address 127.0.98.2:44897)
0928 04:55:40.982156 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 04:55:40.982166 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:55:40.982237 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:55:40.982402 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:55:40.982403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:55:40.982404 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:55:40.982471 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:55:40.982472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.983769 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.983770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:55:40.984947 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:55:40.984949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:55:40.985832 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:55:40.985833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:55:40.986014 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:40.986019 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:55:40.986284 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 04:55:40.986330 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":78,"threads_started":1}
I20260928 04:55:40.986577 11890 leader_election.cc:302] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd038069ac49c5872cd33a75ca26f0, fdd99ac38b684d6795804cbbecff1bf8; no voters:
I20260928 04:55:40.986721 12163 raft_consensus.cc:2781] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:55:40.986781 12163 raft_consensus.cc:482] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:55:40.986810 12163 raft_consensus.cc:3037] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:55:40.986876 12090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1cdef4ad8334bf7bc731ae7743be10b" candidate_uuid: "9ecd038069ac49c5872cd33a75ca26f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93cfc647c10a4315a688fdf059f6514a" is_pre_election: true
I20260928 04:55:40.987032 12090 raft_consensus.cc:2454] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd038069ac49c5872cd33a75ca26f0 in term 0.
I20260928 04:55:40.987746 12163 raft_consensus.cc:504] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cfc647c10a4315a688fdf059f6514a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } } peers { permanent_uuid: "9ecd038069ac49c5872cd33a75ca26f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34311 } } peers { permanent_uuid: "fdd99ac38b684d6795804cbbecff1bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42465 } }
I20260928 04:55:40.987890 12163 leader_election.cc:288] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [CANDIDATE]: Term 1 election: Requested vote from peers 93cfc647c10a4315a688fdf059f6514a (127.0.98.3:37909), fdd99ac38b684d6795804cbbecff1bf8 (127.0.98.1:42465)
I20260928 04:55:40.988041 11818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1cdef4ad8334bf7bc731ae7743be10b" candidate_uuid: "9ecd038069ac49c5872cd33a75ca26f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd99ac38b684d6795804cbbecff1bf8"
I20260928 04:55:40.988092 12090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1cdef4ad8334bf7bc731ae7743be10b" candidate_uuid: "9ecd038069ac49c5872cd33a75ca26f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93cfc647c10a4315a688fdf059f6514a"
I20260928 04:55:40.988135 11818 raft_consensus.cc:3037] T c1cdef4ad8334bf7bc731ae7743be10b P fdd99ac38b684d6795804cbbecff1bf8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:55:40.988157 12090 raft_consensus.cc:3037] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:55:40.988989 12090 raft_consensus.cc:2454] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd038069ac49c5872cd33a75ca26f0 in term 1.
I20260928 04:55:40.988989 11818 raft_consensus.cc:2454] T c1cdef4ad8334bf7bc731ae7743be10b P fdd99ac38b684d6795804cbbecff1bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd038069ac49c5872cd33a75ca26f0 in term 1.
I20260928 04:55:40.989183 11890 leader_election.cc:302] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd038069ac49c5872cd33a75ca26f0, fdd99ac38b684d6795804cbbecff1bf8; no voters:
I20260928 04:55:40.989274 12163 raft_consensus.cc:2781] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:55:40.989332 12163 raft_consensus.cc:686] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd038069ac49c5872cd33a75ca26f0, State: Running, Role: LEADER
I20260928 04:55:40.989414 12163 consensus_queue.cc:237] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cfc647c10a4315a688fdf059f6514a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } } peers { permanent_uuid: "9ecd038069ac49c5872cd33a75ca26f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34311 } } peers { permanent_uuid: "fdd99ac38b684d6795804cbbecff1bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42465 } }
I20260928 04:55:40.990650 11652 catalog_manager.cc:5495] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd038069ac49c5872cd33a75ca26f0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9ecd038069ac49c5872cd33a75ca26f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cfc647c10a4315a688fdf059f6514a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ecd038069ac49c5872cd33a75ca26f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdd99ac38b684d6795804cbbecff1bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20260928 04:55:41.003902 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:41.000431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.0.98.2:34311)
0928 04:55:41.000477 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:55:41.000479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:55:41.000489 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:55:41.000564 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:55:41.000565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:55:41.000586 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:55:41.000674 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:55:41.000840 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:41.001620 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:41.002814 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:55:41.003129 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:55:41.003191 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:55:41.003399 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:55:41.003553 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:55:41.003626 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:55:41.003798 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 04:55:41.003816 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:55:41.006809 11818 raft_consensus.cc:1261] T c1cdef4ad8334bf7bc731ae7743be10b P fdd99ac38b684d6795804cbbecff1bf8 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd038069ac49c5872cd33a75ca26f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:55:41.007364 12090 raft_consensus.cc:1261] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a [term 1 FOLLOWER]: Refusing update from remote peer 9ecd038069ac49c5872cd33a75ca26f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:55:41.007642 12164 consensus_queue.cc:1035] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdd99ac38b684d6795804cbbecff1bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:55:41.007814 12163 consensus_queue.cc:1035] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93cfc647c10a4315a688fdf059f6514a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:55:41.013592 12168 mvcc.cc:205] Tried to move back new op lower bound from 7334180212760760320 to 7334180212693241856. Current Snapshot: MvccSnapshot[applied={T|T < 7334180212760760320}]
I20260928 04:55:41.013739 12170 mvcc.cc:205] Tried to move back new op lower bound from 7334180212760760320 to 7334180212693241856. Current Snapshot: MvccSnapshot[applied={T|T < 7334180212760760320}]
I20260928 04:55:41.013896 12169 mvcc.cc:205] Tried to move back new op lower bound from 7334180212760760320 to 7334180212693241856. Current Snapshot: MvccSnapshot[applied={T|T < 7334180212760760320}]
I20260928 04:55:41.017848 11651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:55:41.017982 11651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:55:41.018853 11651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 350c92cbf80e44db8f0b679c4b103a92: Sending DeleteTablet for 3 replicas of tablet c1cdef4ad8334bf7bc731ae7743be10b
I20260928 04:55:41.019300 11923 tablet_service.cc:1514] Processing DeleteTablet for tablet c1cdef4ad8334bf7bc731ae7743be10b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:55:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39862
I20260928 04:55:41.019614 11791 tablet_service.cc:1514] Processing DeleteTablet for tablet c1cdef4ad8334bf7bc731ae7743be10b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:55:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48792
I20260928 04:55:41.020056 12070 tablet_service.cc:1514] Processing DeleteTablet for tablet c1cdef4ad8334bf7bc731ae7743be10b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:55:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55120
I20260928 04:55:41.020361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 11733
I20260928 04:55:41.025326 12178 tablet_replica.cc:330] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0: stopping tablet replica
I20260928 04:55:41.025424 12178 raft_consensus.cc:2229] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:55:41.025537 12178 raft_consensus.cc:2258] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:55:41.026095 12178 ts_tablet_manager.cc:1905] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:55:41.028182 12178 ts_tablet_manager.cc:1918] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:55:41.028272 12178 log.cc:1196] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/c1cdef4ad8334bf7bc731ae7743be10b
I20260928 04:55:41.028581 12178 ts_tablet_manager.cc:1939] T c1cdef4ad8334bf7bc731ae7743be10b P 9ecd038069ac49c5872cd33a75ca26f0: Deleting consensus metadata
I20260928 04:55:41.029083 11639 catalog_manager.cc:4841] TS 9ecd038069ac49c5872cd33a75ca26f0 (127.0.98.2:34311): tablet c1cdef4ad8334bf7bc731ae7743be10b (table test-table [id=be721f4d8b95453587d2e93c1261276e]) successfully deleted
W20260928 04:55:41.034260 11638 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42465 (error 108)
I20260928 04:55:41.034318 12180 tablet_replica.cc:330] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a: stopping tablet replica
I20260928 04:55:41.034508 12180 raft_consensus.cc:2229] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:55:41.034654 12180 raft_consensus.cc:2258] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:55:41.035010 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:55:41.034891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42465 (local address 127.0.0.1:48796)
0928 04:55:41.034942 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 04:55:41.034974 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 04:55:41.035142 11638 catalog_manager.cc:4584] TS fdd99ac38b684d6795804cbbecff1bf8 (127.0.98.1:42465): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c1cdef4ad8334bf7bc731ae7743be10b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42465: connect: Connection refused (error 111)
I20260928 04:55:41.035205 12180 ts_tablet_manager.cc:1905] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:55:41.037204 12180 ts_tablet_manager.cc:1918] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:55:41.037287 12180 log.cc:1196] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/c1cdef4ad8334bf7bc731ae7743be10b
I20260928 04:55:41.037902 12180 ts_tablet_manager.cc:1939] T c1cdef4ad8334bf7bc731ae7743be10b P 93cfc647c10a4315a688fdf059f6514a: Deleting consensus metadata
I20260928 04:55:41.038388 11636 catalog_manager.cc:4841] TS 93cfc647c10a4315a688fdf059f6514a (127.0.98.3:37909): tablet c1cdef4ad8334bf7bc731ae7743be10b (table test-table [id=be721f4d8b95453587d2e93c1261276e]) successfully deleted
I20260928 04:55:41.042698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 11870
I20260928 04:55:41.049399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 12007
I20260928 04:55:41.055756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 11620
I20260928 04:55:41.061739 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 04:55:41.406751 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 04:55:41.406876 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 04:55:41.754570 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 04:55:41.754 UTC [11206] LOG: received smart shutdown request
2026-09-28 04:55:41.759 UTC [11206] LOG: background worker "logical replication launcher" (PID 11215) exited with exit code 1
2026-09-28 04:55:41.759 UTC [11210] LOG: shutting down
2026-09-28 04:55:41.992 UTC [11206] LOG: database system is shut down
2026-09-28T04:55:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49140 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:55:42 dist-test-slave-m39l krb5kdc[12190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:55:42 dist-test-slave-m39l krb5kdc[12190](info): set up 2 sockets
Sep 28 04:55:42 dist-test-slave-m39l krb5kdc[12190](info): commencing operation
krb5kdc: starting...
W20260928 04:55:43.158653 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:55:43 dist-test-slave-m39l krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:55:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:55:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 04:55:43.341698 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwNdUqW/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/postgres -l logfile start
2026-09-28 04:55:46.335 UTC [12220] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 04:55:46.335 UTC [12220] LOG: listening on IPv4 address "127.0.98.20", port 47581
2026-09-28 04:55:46.340 UTC [12220] LOG: listening on Unix socket "/tmp/.s.PGSQL.47581"
2026-09-28 04:55:46.362 UTC [12223] LOG: database system was shut down at 2026-09-28 04:55:44 UTC
2026-09-28 04:55:46.366 UTC [12220] LOG: database system is ready to accept connections
I20260928 04:55:47.382541 392 mini_postgres.cc:96] Postgres bound to 47581
2026-09-28 04:55:47.388 UTC [12231] FATAL: database "slave" does not exist
127.0.98.20:47581 - accepting connections
I20260928 04:55:47.389379 392 mini_ranger.cc:161] Starting Ranger...
I20260928 04:55:47.404899 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 04:55:48.063648 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 04:55:48.063743 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-admin
2026-09-28 04:55:48,341 [I] DB FLAVOR :POSTGRES
2026-09-28 04:55:48,342 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 04:55:48,342 [I] Checking connection..
2026-09-28 04:55:48,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 04:55:48,622 [I] Checking connection passed.
2026-09-28 04:55:48,622 [I] --------- Verifying version history table ---------
2026-09-28 04:55:48,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 04:55:48,912 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 04:55:48,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 04:55:49,225 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 04:55:49,226 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 04:55:49,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 04:55:49.491 UTC [12295] WARNING: there is no transaction in progress
2026-09-28 04:55:49,498 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 04:55:49,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 04:55:49,781 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 04:55:49,781 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 04:55:49,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 04:55:50,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 04:55:50,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m39l.c.gcp-upstream.internal','N') ;"
2026-09-28 04:55:50,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 04:55:50,814 [I] Table x_portal_user does not exist in database ranger
2026-09-28 04:55:50,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 04:55:51,076 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 04:55:51,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 04:55:51,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 04:55:51,585 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 04:55:51,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 04:55:52.731 UTC [12403] WARNING: there is no transaction in progress
2026-09-28 04:55:52.752 UTC [12403] WARNING: there is no transaction in progress
2026-09-28 04:55:52.772 UTC [12403] WARNING: there is no transaction in progress
2026-09-28 04:55:52.792 UTC [12403] WARNING: there is no transaction in progress
2026-09-28 04:55:52.811 UTC [12403] WARNING: there is no transaction in progress
2026-09-28 04:55:52.834 UTC [12403] WARNING: there is no transaction in progress
2026-09-28 04:55:52.901 UTC [12403] WARNING: there is no transaction in progress
2026-09-28 04:55:52.910 UTC [12403] WARNING: there is no transaction in progress
2026-09-28 04:55:52.929 UTC [12403] WARNING: there is no transaction in progress
2026-09-28 04:55:52.942 UTC [12403] WARNING: there is no transaction in progress
2026-09-28 04:55:52.955 UTC [12403] WARNING: there is no transaction in progress
2026-09-28 04:55:52.974 UTC [12403] WARNING: there is no transaction in progress
2026-09-28 04:55:52.986 UTC [12403] WARNING: there is no transaction in progress
2026-09-28 04:55:52.995 UTC [12403] WARNING: there is no transaction in progress
2026-09-28 04:55:53.004 UTC [12403] WARNING: there is no transaction in progress
2026-09-28 04:55:53.015 UTC [12403] WARNING: there is no transaction in progress
2026-09-28 04:55:53,902 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 04:55:53,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 04:55:54,164 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 04:55:54,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 04:55:54,430 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 04:55:54,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 04:55:54,705 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 04:55:54,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 04:55:55,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 04:55:55,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m39l.c.gcp-upstream.internal';"
2026-09-28 04:55:55,547 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 04:55:55,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 04:55:55,813 [I] DB_PATCHES have already been applied
I20260928 04:55:55.819932 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwNdUqW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 04:55:55.820039 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 04:55:55.823654 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:37113
Sep 28, 2026 4:55:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:55:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:55:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 4:55:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 4:55:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 4:55:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 4:55:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 04:55:56.988708 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.599s user 0.004s sys 0.005s
Sep 28, 2026 4:56:19 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 4:56:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 4:56:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 4:56:19 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 4:56:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 4:56:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 4:56:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 4:56:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 4:56:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 4:56:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 4:56:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 4:56:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 4:56:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 4:56:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 4:56:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 4:56:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 4:56:19 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 4:56:20 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 04:56:20.760715 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 04:56:20 dist-test-slave-m39l krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571380, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 04:56:20.864873 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 04:56:20.876238 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 04:56:21.864042 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 04:56:21.864141 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-kms
I20260928 04:56:21.864162 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:47581
2026-09-28 04:56:22,039 [I] DB FLAVOR :POSTGRES
2026-09-28 04:56:22,039 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 04:56:22,039 [I] Checking connection
2026-09-28 04:56:22,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 04:56:22,324 [I] connection success
2026-09-28 04:56:22,325 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 04:56:22,325 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 04:56:22,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 04:56:22,645 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 04:56:22,645 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 04:56:22,645 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 04:56:22,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47581/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 04:56:22,958 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 04:56:23.492684 392 mini_ranger_kms.cc:310] Created kms service
I20260928 04:56:23.693410 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 04:56:23.747627 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 04:56:23 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571380, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:56:24.185818 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 04:56:24.186033 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwNdUqW/build/release/bin/postgresql.jar:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwNdUqW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 04:56:24.186057 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 04:56:24.190011 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12584
I20260928 04:56:24.190109 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:48345
Sep 28, 2026 4:56:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:56:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:56:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 4:56:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 4:56:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 4:56:25 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 04:56:26.551007 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.686s user 0.007s sys 0.146s
I20260928 04:56:26.551113 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 04:56:26.551175 392 mini_ranger_kms.cc:401] 127.0.98.20:48345/kms/v1/keys
Sep 28 04:56:26 dist-test-slave-m39l krb5kdc[12190](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790571386, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 04:56:27 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571386, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 04:56:27 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571386, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 04:56:29 dist-test-slave-m39l krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:56:29.721307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38663
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38663
--ranger_config_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48345/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:56:29.804493 12634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:29.804740 12634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:29.804778 12634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:29.806048 12634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:56:29.806111 12634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:29.806136 12634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:29.806167 12634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:56:29.806193 12634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:56:29.807828 12634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:48345/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38663
--ranger_config_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:29.808106 12634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:29.808327 12634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:29.813230 12642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:29.813292 12640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:29.813218 12639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:29.814059 12634 server_base.cc:1034] running on GCE node
I20260928 04:56:29.814332 12634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:29.814702 12634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:29.815896 12634 hybrid_clock.cc:648] HybridClock initialized: now 1790571389815862 us; error 46 us; skew 500 ppm
Sep 28 04:56:29 dist-test-slave-m39l krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:29.826401 12634 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:56:29.826956 12634 webserver.cc:460] Webserver started at http://127.0.98.62:40129/ using document root <none> and password file <none>
I20260928 04:56:29.827150 12634 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:29.827188 12634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:29.827281 12634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:56:29 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:56:29.903391 12634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "c32927b9dd3e4d54bba766e8c6f30677"
format_stamp: "Formatted at 2026-09-28 04:56:29 on dist-test-slave-m39l"
server_key: "d74eb4d6cfed2c34bfdbe2cde7f60069"
server_key_iv: "55209403bfa6a1c202d96097e9ea2176"
server_key_version: "kuduclusterkey@0"
I20260928 04:56:29.903960 12634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "c32927b9dd3e4d54bba766e8c6f30677"
format_stamp: "Formatted at 2026-09-28 04:56:29 on dist-test-slave-m39l"
server_key: "d74eb4d6cfed2c34bfdbe2cde7f60069"
server_key_iv: "55209403bfa6a1c202d96097e9ea2176"
server_key_version: "kuduclusterkey@0"
I20260928 04:56:29.905973 12634 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 04:56:29.921288 12649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:29.921779 12634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:56:29.921881 12634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "c32927b9dd3e4d54bba766e8c6f30677"
format_stamp: "Formatted at 2026-09-28 04:56:29 on dist-test-slave-m39l"
server_key: "d74eb4d6cfed2c34bfdbe2cde7f60069"
server_key_iv: "55209403bfa6a1c202d96097e9ea2176"
server_key_version: "kuduclusterkey@0"
I20260928 04:56:29.921977 12634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:29.950152 12634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:29.952560 12634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:29.952735 12634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:29.957952 12634 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38663
I20260928 04:56:29.957983 12701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38663 every 8 connection(s)
I20260928 04:56:29.958415 12634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:56:29.959204 12702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:29.961597 12702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677: Bootstrap starting.
I20260928 04:56:29.962212 12702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:29.962564 12702 log.cc:824] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:29.963313 12702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677: No bootstrap required, opened a new log
I20260928 04:56:29.965507 12702 raft_consensus.cc:348] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32927b9dd3e4d54bba766e8c6f30677" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38663 } }
I20260928 04:56:29.965699 12702 raft_consensus.cc:374] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:29.965770 12702 raft_consensus.cc:729] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c32927b9dd3e4d54bba766e8c6f30677, State: Initialized, Role: FOLLOWER
I20260928 04:56:29.965956 12702 consensus_queue.cc:260] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32927b9dd3e4d54bba766e8c6f30677" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38663 } }
I20260928 04:56:29.966058 12702 raft_consensus.cc:388] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:56:29.966101 12702 raft_consensus.cc:482] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:56:29.966154 12702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:29.966846 12702 raft_consensus.cc:504] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32927b9dd3e4d54bba766e8c6f30677" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38663 } }
I20260928 04:56:29.966969 12702 leader_election.cc:302] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c32927b9dd3e4d54bba766e8c6f30677; no voters:
I20260928 04:56:29.967185 12702 leader_election.cc:288] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:56:29.967213 12705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:56:29.967332 12705 raft_consensus.cc:686] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [term 1 LEADER]: Becoming Leader. State: Replica: c32927b9dd3e4d54bba766e8c6f30677, State: Running, Role: LEADER
I20260928 04:56:29.967490 12702 sys_catalog.cc:564] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:56:29.967497 12705 consensus_queue.cc:237] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32927b9dd3e4d54bba766e8c6f30677" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38663 } }
I20260928 04:56:29.967957 12706 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c32927b9dd3e4d54bba766e8c6f30677" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32927b9dd3e4d54bba766e8c6f30677" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38663 } } }
I20260928 04:56:29.968060 12706 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [sys.catalog]: This master's current role is: LEADER
I20260928 04:56:29.968484 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 12634
I20260928 04:56:29.968585 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:56:29.968815 392 external_mini_cluster.cc:1428] Setting key fd649efce5c7061e95f1c8e7cddc2a43
I20260928 04:56:29.969094 12702 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 04:56:29.969261 12707 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c32927b9dd3e4d54bba766e8c6f30677. Latest consensus state: current_term: 1 leader_uuid: "c32927b9dd3e4d54bba766e8c6f30677" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32927b9dd3e4d54bba766e8c6f30677" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38663 } } }
I20260928 04:56:29.969348 12707 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [sys.catalog]: This master's current role is: LEADER
I20260928 04:56:29.969509 12712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:56:29.970012 12712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 04:56:29 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:29.983235 12712 catalog_manager.cc:1347] Generated new cluster ID: d70938b4a1864ec9b28f092be70bbf75
I20260928 04:56:29.983325 12712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:56:29.995126 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:29.969614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.0.98.62:38663)
0928 04:56:29.970792 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:29.970802 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:29.970840 (+ 38us) server_negotiation.cc:407] Connection header received
0928 04:56:29.970956 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:29.970963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:29.971020 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:29.971293 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:29.972735 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:29.976874 (+ 4139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:29.978238 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:29.979017 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:29.979157 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:29.983065 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:29.983072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:29.983072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:29.983089 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:29.992559 (+ 9470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:29.993147 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:29.993150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:29.993151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:29.993276 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:29.993540 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:29.993540 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:29.993541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:29.993935 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:29.994111 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:29.994690 (+ 579us) server_negotiation.cc:299] Negotiation successful
0928 04:56:29.994842 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1129,"thread_start_us":74,"threads_started":1}
I20260928 04:56:30.005093 12712 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:56:30.005861 12712 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:56:30.013440 12712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677: Generated new TSK 0
I20260928 04:56:30.013644 12712 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:56:30.734320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38663
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:56:30.847736 12747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:30.847957 12747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:30.847983 12747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:30.849324 12747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:30.849381 12747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:30.849411 12747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:56:30.851318 12747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:48345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:30.851600 12747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:30.851866 12747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:30.858343 12755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:30.858280 12753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:30.858278 12752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:30.858726 12747 server_base.cc:1034] running on GCE node
I20260928 04:56:30.858983 12747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:30.859320 12747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:30.860492 12747 hybrid_clock.cc:648] HybridClock initialized: now 1790571390860469 us; error 40 us; skew 500 ppm
Sep 28 04:56:30 dist-test-slave-m39l krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:30.863653 12747 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:56:30.864244 12747 webserver.cc:460] Webserver started at http://127.0.98.1:35659/ using document root <none> and password file <none>
I20260928 04:56:30.864610 12747 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:30.864713 12747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:30.864866 12747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:56:30 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:56:30.878402 12747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "b6772fda83ae4260a6b240656b6aafe0"
format_stamp: "Formatted at 2026-09-28 04:56:30 on dist-test-slave-m39l"
server_key: "bfcac232bce07e5a5eac140ffbac2bee"
server_key_iv: "b5352c3fed6c2daeba23e74874b50941"
server_key_version: "kuduclusterkey@0"
I20260928 04:56:30.878933 12747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b6772fda83ae4260a6b240656b6aafe0"
format_stamp: "Formatted at 2026-09-28 04:56:30 on dist-test-slave-m39l"
server_key: "bfcac232bce07e5a5eac140ffbac2bee"
server_key_iv: "b5352c3fed6c2daeba23e74874b50941"
server_key_version: "kuduclusterkey@0"
I20260928 04:56:30.882616 12747 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 04:56:30.903098 12764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:30.903430 12747 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260928 04:56:30.903529 12747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "b6772fda83ae4260a6b240656b6aafe0"
format_stamp: "Formatted at 2026-09-28 04:56:30 on dist-test-slave-m39l"
server_key: "bfcac232bce07e5a5eac140ffbac2bee"
server_key_iv: "b5352c3fed6c2daeba23e74874b50941"
server_key_version: "kuduclusterkey@0"
I20260928 04:56:30.903630 12747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:30.932812 12747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:30.935206 12747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:30.935367 12747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:30.935653 12747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:56:30.936039 12747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:56:30.936090 12747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:30.936164 12747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:56:30.936208 12747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:30.943697 12747 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33273
I20260928 04:56:30.943748 12877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33273 every 8 connection(s)
I20260928 04:56:30.944263 12747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:56:30.951804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 12747
I20260928 04:56:30.951917 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:56:30.952093 392 external_mini_cluster.cc:1428] Setting key 95e0e81896ca547074863e25d18601c4
Sep 28 04:56:30 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:30.965986 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:30.945141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38663 (local address 127.0.98.1:40167)
0928 04:56:30.950906 (+ 5765us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:30.950977 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:30.951701 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:30.952231 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:30.952236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:30.952996 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:30.953304 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:30.953311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:30.954338 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:30.954339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:30.955829 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:30.955832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:30.956417 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:30.956418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:30.956558 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:30.957203 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:30.957215 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:30.958889 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:30.964185 (+ 5296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:30.964193 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:30.964196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:30.964542 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:30.964785 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:30.964785 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:30.964786 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:30.964933 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:30.965268 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:30.965274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:30.965473 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 04:56:30.965579 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 04:56:30.965750 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5712,"thread_start_us":87,"threads_started":1}
I20260928 04:56:30.966476 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:30.945148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40167 (local address 127.0.98.62:38663)
0928 04:56:30.951931 (+ 6783us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:30.951934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:30.951943 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:56:30.951971 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:30.951972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:30.951987 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:30.952056 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:30.953419 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:30.954224 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:30.955932 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:30.956269 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:30.956334 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:30.959013 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:30.959015 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:30.959015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:30.959024 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:30.964055 (+ 5031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:30.964640 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:30.964641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:30.964641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:30.964711 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:30.965008 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:30.965009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:30.965010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:30.965170 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:30.965238 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:30.966288 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0928 04:56:30.966380 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6745,"thread_start_us":87,"threads_started":1}
I20260928 04:56:30.966986 12878 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38663
I20260928 04:56:30.967085 12878 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:30.967295 12878 heartbeater.cc:511] Master 127.0.98.62:38663 requested a full tablet report, sending...
I20260928 04:56:30.967864 12666 ts_manager.cc:194] Registered new tserver with Master: b6772fda83ae4260a6b240656b6aafe0 (127.0.98.1:33273)
I20260928 04:56:30.968775 12666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40167
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:56:31.001578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38663
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:56:31 dist-test-slave-m39l krb5kdc[12190](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790571391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 04:56:31.100528 12886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:31.100942 12886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:31.101027 12886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:31.103011 12886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:31.103170 12886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:31.103263 12886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:56:31.105835 12886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:48345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:31.106377 12886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:31.106791 12886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:31.114085 12892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:31.114122 12894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:31.114480 12891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:31.114688 12886 server_base.cc:1034] running on GCE node
I20260928 04:56:31.114984 12886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:31.115298 12886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:31.116446 12886 hybrid_clock.cc:648] HybridClock initialized: now 1790571391116444 us; error 52 us; skew 500 ppm
Sep 28 04:56:31 dist-test-slave-m39l krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:31.119266 12886 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:56:31.119791 12886 webserver.cc:460] Webserver started at http://127.0.98.2:36187/ using document root <none> and password file <none>
I20260928 04:56:31.120036 12886 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:31.120105 12886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:31.120258 12886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:56:31 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:56:31.134243 12886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "11a4f8c40a8d404c984c36453a8f0b6e"
format_stamp: "Formatted at 2026-09-28 04:56:31 on dist-test-slave-m39l"
server_key: "ba0a64063745b2a8978e5372aca4186d"
server_key_iv: "8a38c54ccd7ed32a626659b55a40e1c6"
server_key_version: "kuduclusterkey@0"
I20260928 04:56:31.134765 12886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "11a4f8c40a8d404c984c36453a8f0b6e"
format_stamp: "Formatted at 2026-09-28 04:56:31 on dist-test-slave-m39l"
server_key: "ba0a64063745b2a8978e5372aca4186d"
server_key_iv: "8a38c54ccd7ed32a626659b55a40e1c6"
server_key_version: "kuduclusterkey@0"
I20260928 04:56:31.136766 12886 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 04:56:31.150048 12901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:31.150319 12886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:31.150431 12886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "11a4f8c40a8d404c984c36453a8f0b6e"
format_stamp: "Formatted at 2026-09-28 04:56:31 on dist-test-slave-m39l"
server_key: "ba0a64063745b2a8978e5372aca4186d"
server_key_iv: "8a38c54ccd7ed32a626659b55a40e1c6"
server_key_version: "kuduclusterkey@0"
I20260928 04:56:31.150563 12886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:31.175628 12886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:31.178299 12886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:31.178562 12886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:31.178884 12886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:56:31.179337 12886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:56:31.179435 12886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:31.179513 12886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:56:31.179579 12886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:31.187947 12886 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46541
I20260928 04:56:31.188549 12886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:56:31.190223 13015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46541 every 8 connection(s)
Sep 28 04:56:31 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:31.198704 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 12886
I20260928 04:56:31.198802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:56:31.198997 392 external_mini_cluster.cc:1428] Setting key 90204e2c1d6f9882bda47958868e3247
I20260928 04:56:31.201969 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:31.190119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39957 (local address 127.0.98.62:38663)
0928 04:56:31.190174 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:31.190177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:31.190591 (+ 414us) server_negotiation.cc:407] Connection header received
0928 04:56:31.191426 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:31.191427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:31.191441 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:31.191474 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:31.192351 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:31.193095 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:31.194633 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:31.194960 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:31.195009 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:31.198235 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:31.198237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:31.198237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:31.198246 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:31.200220 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:31.200756 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:31.200757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:31.200757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:31.200819 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:31.201133 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:31.201134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:31.201134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:31.201283 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:31.201327 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:31.201764 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 04:56:31.201868 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:56:31.202807 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:31.189849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38663 (local address 127.0.98.2:39957)
0928 04:56:31.190437 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:31.190479 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:31.191295 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:31.191665 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:31.191670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:31.191932 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:31.192238 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:31.192246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:31.193228 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:31.193229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:31.194528 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:31.194531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:31.195116 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:31.195117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:31.195273 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:31.196320 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:31.196331 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:31.198096 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:31.200326 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:31.200334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:31.200336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:31.200659 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:31.200898 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:31.200898 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:31.200899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:31.201049 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:31.201390 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:31.201396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:31.201588 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 04:56:31.202346 (+ 758us) client_negotiation.cc:240] Negotiation successful
0928 04:56:31.202561 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":78,"threads_started":1}
I20260928 04:56:31.205088 13016 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38663
I20260928 04:56:31.205201 13016 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:31.205391 13016 heartbeater.cc:511] Master 127.0.98.62:38663 requested a full tablet report, sending...
I20260928 04:56:31.205765 12666 ts_manager.cc:194] Registered new tserver with Master: 11a4f8c40a8d404c984c36453a8f0b6e (127.0.98.2:46541)
I20260928 04:56:31.206292 12666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39957
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:56:31.230603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38663
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:56:31.338394 13021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:31.338825 13021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:31.338922 13021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:31.340770 13021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:31.340845 13021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:31.340893 13021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:56:31.343155 13021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:48345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:31.343442 13021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:31.343705 13021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:56:31.351015 13021 server_base.cc:1034] running on GCE node
W20260928 04:56:31.350986 13026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:31.351115 13027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:31.350982 13029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:31.351608 13021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:31.351913 13021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:31.353091 13021 hybrid_clock.cc:648] HybridClock initialized: now 1790571391353067 us; error 39 us; skew 500 ppm
Sep 28 04:56:31 dist-test-slave-m39l krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:31.356076 13021 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:56:31.356632 13021 webserver.cc:460] Webserver started at http://127.0.98.3:41043/ using document root <none> and password file <none>
I20260928 04:56:31.356891 13021 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:31.356950 13021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:31.357097 13021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:56:31 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:56:31.374315 13021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "8cfe4f8bce6a49e7bc94e16b01c55b25"
format_stamp: "Formatted at 2026-09-28 04:56:31 on dist-test-slave-m39l"
server_key: "0b9ab25a138762baaf6326ffd7775413"
server_key_iv: "a867e9653a6b6b84945086fe396f8d52"
server_key_version: "kuduclusterkey@0"
I20260928 04:56:31.374845 13021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8cfe4f8bce6a49e7bc94e16b01c55b25"
format_stamp: "Formatted at 2026-09-28 04:56:31 on dist-test-slave-m39l"
server_key: "0b9ab25a138762baaf6326ffd7775413"
server_key_iv: "a867e9653a6b6b84945086fe396f8d52"
server_key_version: "kuduclusterkey@0"
I20260928 04:56:31.376735 13021 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 04:56:31.393167 13036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:31.393457 13021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:31.393580 13021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "8cfe4f8bce6a49e7bc94e16b01c55b25"
format_stamp: "Formatted at 2026-09-28 04:56:31 on dist-test-slave-m39l"
server_key: "0b9ab25a138762baaf6326ffd7775413"
server_key_iv: "a867e9653a6b6b84945086fe396f8d52"
server_key_version: "kuduclusterkey@0"
I20260928 04:56:31.393709 13021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:31.410794 13021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:31.413573 13021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:31.413802 13021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:31.414125 13021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:56:31.414597 13021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:56:31.414691 13021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:31.414770 13021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:56:31.414829 13021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:31.421849 13021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46223
I20260928 04:56:31.422391 13021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:56:31.424233 13149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46223 every 8 connection(s)
I20260928 04:56:31.427587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 13021
I20260928 04:56:31.427686 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:56:31.427877 392 external_mini_cluster.cc:1428] Setting key 21b0987039ad489085490cd5fd5d7e39
Sep 28 04:56:31 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:31.435037 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:31.424938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48945 (local address 127.0.98.62:38663)
0928 04:56:31.424997 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:31.424999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:31.425242 (+ 243us) server_negotiation.cc:407] Connection header received
0928 04:56:31.425861 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:31.425863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:31.425876 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:31.425932 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:31.426758 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:31.427386 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:31.428833 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:31.429146 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:31.429220 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:31.431807 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:31.431808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:31.431809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:31.431817 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:31.433512 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:31.433969 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:31.433970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:31.433970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:31.434030 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:31.434320 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:31.434321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:31.434321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:31.434447 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:31.434553 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:31.434859 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 04:56:31.434943 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:56:31.435316 13152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:31.424944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38663 (local address 127.0.98.3:48945)
0928 04:56:31.425096 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:31.425139 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:31.425733 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:31.426036 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:31.426041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:31.426307 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:31.426558 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:31.426638 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:31.427496 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:31.427497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:31.428771 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:31.428773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:31.429251 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:31.429253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:31.429382 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:31.429952 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:31.429963 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:31.431690 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:31.433620 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:31.433629 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:31.433632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:31.433890 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:31.434101 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:31.434101 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:31.434102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:31.434259 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:31.434542 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:31.434549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:31.434726 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 04:56:31.434841 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 04:56:31.434999 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260928 04:56:31.435962 13150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38663
I20260928 04:56:31.436082 13150 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:31.436269 13150 heartbeater.cc:511] Master 127.0.98.62:38663 requested a full tablet report, sending...
I20260928 04:56:31.436657 12666 ts_manager.cc:194] Registered new tserver with Master: 8cfe4f8bce6a49e7bc94e16b01c55b25 (127.0.98.3:46223)
I20260928 04:56:31.437031 12666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48945
I20260928 04:56:31.439533 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:56:31.440135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 12747
I20260928 04:56:31.447069 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 12886
I20260928 04:56:31.452865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 13021
I20260928 04:56:31.459002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 12634
I20260928 04:56:31.464192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38663
--webserver_interface=127.0.98.62
--webserver_port=40129
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38663
--ranger_config_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48345/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48345/kms with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:56:31.547991 13153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:31.548277 13153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:31.548321 13153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:31.549685 13153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:56:31.549777 13153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:31.549798 13153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:31.549818 13153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:56:31.549844 13153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:56:31.551702 13153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48345/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38663
--ranger_config_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:31.552021 13153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:31.552266 13153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:31.557611 13159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:31.557649 13161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:31.557619 13158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:31.557782 13153 server_base.cc:1034] running on GCE node
I20260928 04:56:31.558138 13153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:31.558442 13153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:31.559638 13153 hybrid_clock.cc:648] HybridClock initialized: now 1790571391559616 us; error 40 us; skew 500 ppm
Sep 28 04:56:31 dist-test-slave-m39l krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:31.562186 13153 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:56:31.562724 13153 webserver.cc:460] Webserver started at http://127.0.98.62:40129/ using document root <none> and password file <none>
I20260928 04:56:31.562965 13153 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:31.563017 13153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 04:56:31 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:56:31.615584 13153 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:31.616381 13169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:31.616569 13153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:31.616662 13153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "c32927b9dd3e4d54bba766e8c6f30677"
format_stamp: "Formatted at 2026-09-28 04:56:29 on dist-test-slave-m39l"
server_key: "d74eb4d6cfed2c34bfdbe2cde7f60069"
server_key_iv: "55209403bfa6a1c202d96097e9ea2176"
server_key_version: "kuduclusterkey@0"
I20260928 04:56:31.617026 13153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:31.644268 13153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:31.646608 13153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:31.646737 13153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:31.651508 13153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38663
I20260928 04:56:31.651548 13221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38663 every 8 connection(s)
I20260928 04:56:31.651980 13153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:56:31.653242 13222 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:56:31.653975 13222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677: Bootstrap starting.
I20260928 04:56:31.660490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 13153
I20260928 04:56:31.660817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33273
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38663
--webserver_port=35659
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48345/kms with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:56:31.661972 13222 log.cc:824] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:31.663975 13222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:56:31.664297 13222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677: Bootstrap complete.
I20260928 04:56:31.667037 13222 raft_consensus.cc:348] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32927b9dd3e4d54bba766e8c6f30677" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38663 } }
I20260928 04:56:31.667310 13222 raft_consensus.cc:729] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c32927b9dd3e4d54bba766e8c6f30677, State: Initialized, Role: FOLLOWER
I20260928 04:56:31.667522 13222 consensus_queue.cc:260] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32927b9dd3e4d54bba766e8c6f30677" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38663 } }
I20260928 04:56:31.667605 13222 raft_consensus.cc:388] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:56:31.667666 13222 raft_consensus.cc:482] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:56:31.667725 13222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [term 1 FOLLOWER]: Advancing to term 2
I20260928 04:56:31.668805 13222 raft_consensus.cc:504] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32927b9dd3e4d54bba766e8c6f30677" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38663 } }
I20260928 04:56:31.668934 13222 leader_election.cc:302] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c32927b9dd3e4d54bba766e8c6f30677; no voters:
I20260928 04:56:31.669147 13222 leader_election.cc:288] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 04:56:31.669178 13226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 04:56:31.669294 13226 raft_consensus.cc:686] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [term 2 LEADER]: Becoming Leader. State: Replica: c32927b9dd3e4d54bba766e8c6f30677, State: Running, Role: LEADER
I20260928 04:56:31.669384 13226 consensus_queue.cc:237] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32927b9dd3e4d54bba766e8c6f30677" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38663 } }
I20260928 04:56:31.669443 13222 sys_catalog.cc:564] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:56:31.669657 13227 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c32927b9dd3e4d54bba766e8c6f30677. Latest consensus state: current_term: 2 leader_uuid: "c32927b9dd3e4d54bba766e8c6f30677" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32927b9dd3e4d54bba766e8c6f30677" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38663 } } }
I20260928 04:56:31.669907 13227 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [sys.catalog]: This master's current role is: LEADER
I20260928 04:56:31.669832 13228 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c32927b9dd3e4d54bba766e8c6f30677" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32927b9dd3e4d54bba766e8c6f30677" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38663 } } }
I20260928 04:56:31.670193 13228 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677 [sys.catalog]: This master's current role is: LEADER
I20260928 04:56:31.670298 13234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:56:31.670874 13234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:56:31.671028 13234 catalog_manager.cc:1259] Loaded cluster ID: d70938b4a1864ec9b28f092be70bbf75
I20260928 04:56:31.671059 13234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:56:31.671237 13222 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 04:56:31.672391 13234 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:56:31.672577 13234 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677: Loaded TSK: 0
I20260928 04:56:31.672812 13234 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 04:56:31.776135 13224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:31.776422 13224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:31.776459 13224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:31.778272 13224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:31.778357 13224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:31.778404 13224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:56:31.780893 13224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35659
--tserver_master_addrs=127.0.98.62:38663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:31.781306 13224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:31.781585 13224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:31.788090 13262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:31.788103 13260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:31.788249 13259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:31.789582 13224 server_base.cc:1034] running on GCE node
I20260928 04:56:31.789865 13224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:31.790174 13224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:31.791360 13224 hybrid_clock.cc:648] HybridClock initialized: now 1790571391791333 us; error 42 us; skew 500 ppm
Sep 28 04:56:31 dist-test-slave-m39l krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:31.794878 13224 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:56:31.795451 13224 webserver.cc:460] Webserver started at http://127.0.98.1:35659/ using document root <none> and password file <none>
I20260928 04:56:31.795823 13224 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:31.795933 13224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 04:56:31 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:56:31.813069 13224 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 04:56:31.818212 13270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:31.818451 13224 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260928 04:56:31.818635 13224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "b6772fda83ae4260a6b240656b6aafe0"
format_stamp: "Formatted at 2026-09-28 04:56:30 on dist-test-slave-m39l"
server_key: "bfcac232bce07e5a5eac140ffbac2bee"
server_key_iv: "b5352c3fed6c2daeba23e74874b50941"
server_key_version: "kuduclusterkey@0"
I20260928 04:56:31.819031 13224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:31.843562 13224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:31.846297 13224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:31.846616 13224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:31.846961 13224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:56:31.847366 13224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:56:31.847424 13224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:31.847470 13224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:56:31.847510 13224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:31.855603 13224 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33273
I20260928 04:56:31.855696 13383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33273 every 8 connection(s)
I20260928 04:56:31.856093 13224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:56:31.857398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 13224
I20260928 04:56:31.857728 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46541
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38663
--webserver_port=36187
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48345/kms with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:56:31 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:31.871688 13386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:31.856968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38663 (local address 127.0.98.1:48785)
0928 04:56:31.857417 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:31.857471 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:31.858279 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:31.859068 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:31.859072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:31.859382 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:31.859714 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:31.859722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:31.861469 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:31.861470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:31.862922 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:31.862926 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:31.863991 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:31.863993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:31.864142 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:31.864807 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:31.864819 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:31.866744 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:31.869364 (+ 2620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:31.869373 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:31.869377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:31.869803 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:31.870138 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:31.870139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:31.870140 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:31.870293 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:31.870929 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:31.870936 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:31.871111 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 04:56:31.871238 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 04:56:31.871413 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":99,"threads_started":1}
I20260928 04:56:31.872632 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:31.857064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48785 (local address 127.0.98.62:38663)
0928 04:56:31.857326 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:31.857336 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:31.857612 (+ 276us) server_negotiation.cc:407] Connection header received
0928 04:56:31.858667 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:31.858675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:31.858740 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:31.858878 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:31.859870 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:31.861324 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:31.863038 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:31.863751 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:31.863871 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:31.866898 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:31.866904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:31.866905 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:31.866920 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:31.869245 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:31.869914 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:31.869917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:31.869918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:31.870044 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:31.870382 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:31.870383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:31.870383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:31.870802 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:31.870896 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:31.872275 (+ 1379us) server_negotiation.cc:299] Negotiation successful
0928 04:56:31.872410 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":106,"threads_started":1}
I20260928 04:56:31.873785 13384 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38663
I20260928 04:56:31.873973 13384 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:31.874228 13384 heartbeater.cc:511] Master 127.0.98.62:38663 requested a full tablet report, sending...
W20260928 04:56:31.875162 13384 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38663 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 04:56:31.875310 13384 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:31.875434 13384 heartbeater.cc:511] Master 127.0.98.62:38663 requested a full tablet report, sending...
I20260928 04:56:31.875901 13384 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:31.876032 13384 heartbeater.cc:511] Master 127.0.98.62:38663 requested a full tablet report, sending...
W20260928 04:56:31.876343 13384 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:56:31.971292 13388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:31.971563 13388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:31.971609 13388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:31.973039 13388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:31.973119 13388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:31.973169 13388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:56:31.975095 13388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36187
--tserver_master_addrs=127.0.98.62:38663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:31.975623 13388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:31.975921 13388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:31.982081 13396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:31.982595 13393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:31.982152 13394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:31.982919 13388 server_base.cc:1034] running on GCE node
I20260928 04:56:31.983136 13388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:31.983419 13388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:31.990784 13388 hybrid_clock.cc:648] HybridClock initialized: now 1790571391990759 us; error 38 us; skew 500 ppm
Sep 28 04:56:31 dist-test-slave-m39l krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:31.994135 13388 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:56:31.994756 13388 webserver.cc:460] Webserver started at http://127.0.98.2:36187/ using document root <none> and password file <none>
I20260928 04:56:31.995024 13388 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:31.995143 13388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 04:56:31 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:56:32.018740 13388 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.000s
I20260928 04:56:32.020146 13404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:32.020424 13388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:32.020533 13388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "11a4f8c40a8d404c984c36453a8f0b6e"
format_stamp: "Formatted at 2026-09-28 04:56:31 on dist-test-slave-m39l"
server_key: "ba0a64063745b2a8978e5372aca4186d"
server_key_iv: "8a38c54ccd7ed32a626659b55a40e1c6"
server_key_version: "kuduclusterkey@0"
I20260928 04:56:32.021006 13388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:32.049747 13388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:32.052634 13388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:32.052819 13388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:32.053088 13388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:56:32.053467 13388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:56:32.053517 13388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:32.053570 13388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:56:32.053596 13388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:32.060314 13388 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46541
I20260928 04:56:32.060412 13517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46541 every 8 connection(s)
I20260928 04:56:32.060752 13388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:56:32.065817 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 13388
I20260928 04:56:32.066125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46223
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38663
--webserver_port=41043
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48345/kms with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:56:32 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:32.073190 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:32.061504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42663 (local address 127.0.98.62:38663)
0928 04:56:32.061553 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:32.061555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:32.061866 (+ 311us) server_negotiation.cc:407] Connection header received
0928 04:56:32.062555 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:32.062557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:32.062569 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:32.062652 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:32.063218 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:32.063724 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:32.064718 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:32.064960 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:32.065054 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:32.067599 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:32.067601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:32.067601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:32.067609 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:32.071665 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:32.072161 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:32.072162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:32.072163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:32.072226 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:32.072492 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:32.072493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:32.072494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:32.072621 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:32.072696 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:32.073001 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 04:56:32.073092 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:56:32.073428 13520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:32.061512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38663 (local address 127.0.98.2:42663)
0928 04:56:32.061716 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:32.061764 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:32.062421 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:32.062732 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:32.062736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:32.062926 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:32.063122 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:32.063126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:32.063818 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:32.063819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:32.064627 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:32.064629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:32.065079 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:32.065080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:32.065190 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:32.065745 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:32.065756 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:32.067459 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:32.071782 (+ 4323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:32.071793 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:32.071796 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:32.072069 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:32.072292 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:32.072293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:32.072293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:32.072424 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:32.072687 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:32.072695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:32.072864 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 04:56:32.072971 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 04:56:32.073145 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
I20260928 04:56:32.073891 13518 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38663
I20260928 04:56:32.074007 13518 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:32.074236 13518 heartbeater.cc:511] Master 127.0.98.62:38663 requested a full tablet report, sending...
W20260928 04:56:32.074918 13518 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38663 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 04:56:32.075001 13518 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:32.075074 13518 heartbeater.cc:511] Master 127.0.98.62:38663 requested a full tablet report, sending...
I20260928 04:56:32.075361 13518 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:32.075433 13518 heartbeater.cc:511] Master 127.0.98.62:38663 requested a full tablet report, sending...
W20260928 04:56:32.075701 13518 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:56:32.193241 13521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:32.193542 13521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:32.193588 13521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:32.195554 13521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:32.195626 13521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:32.195670 13521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:56:32.198170 13521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41043
--tserver_master_addrs=127.0.98.62:38663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:32.198518 13521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:32.198825 13521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:32.204872 13527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:32.205166 13521 server_base.cc:1034] running on GCE node
W20260928 04:56:32.204890 13530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:32.205488 13528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:32.205725 13521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:32.206032 13521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:32.207224 13521 hybrid_clock.cc:648] HybridClock initialized: now 1790571392207207 us; error 32 us; skew 500 ppm
Sep 28 04:56:32 dist-test-slave-m39l krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:32.210628 13521 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:56:32.211292 13521 webserver.cc:460] Webserver started at http://127.0.98.3:41043/ using document root <none> and password file <none>
I20260928 04:56:32.211601 13521 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:32.211699 13521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 04:56:32 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:56:32.235172 13521 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260928 04:56:32.236143 13538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:32.236377 13521 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:32.236505 13521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "8cfe4f8bce6a49e7bc94e16b01c55b25"
format_stamp: "Formatted at 2026-09-28 04:56:31 on dist-test-slave-m39l"
server_key: "0b9ab25a138762baaf6326ffd7775413"
server_key_iv: "a867e9653a6b6b84945086fe396f8d52"
server_key_version: "kuduclusterkey@0"
I20260928 04:56:32.236923 13521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:32.263214 13521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:32.265616 13521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:32.265805 13521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:32.266065 13521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:56:32.266467 13521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:56:32.266554 13521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:32.266602 13521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:56:32.266633 13521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:32.274466 13521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46223
I20260928 04:56:32.274571 13651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46223 every 8 connection(s)
I20260928 04:56:32.275080 13521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:56:32.283636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 13521
Sep 28 04:56:32 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:32.292162 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:32.280785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46609 (local address 127.0.98.62:38663)
0928 04:56:32.280838 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:32.280840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:32.281133 (+ 293us) server_negotiation.cc:407] Connection header received
0928 04:56:32.281734 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:32.281736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:32.281749 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:32.281836 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:32.282425 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:32.283247 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:32.285219 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:32.285498 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:32.285565 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:32.288227 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:32.288229 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:32.288230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:32.288237 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:32.290143 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:32.290987 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:32.290988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:32.290989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:32.291046 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:32.291385 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:32.291386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:32.291386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:32.291524 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:32.291558 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:32.291967 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 04:56:32.292054 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:56:32.292414 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:32.280803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38663 (local address 127.0.98.3:46609)
0928 04:56:32.280981 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:32.281028 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:32.281620 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:32.281898 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:32.281901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:32.282095 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:32.282308 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:32.282312 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:32.283370 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:32.283372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:32.285101 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:32.285104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:32.285618 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:32.285620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:32.285780 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:32.286420 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:32.286432 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:32.288131 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:32.290208 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:32.290216 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:32.290218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:32.290888 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:32.291126 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:32.291126 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:32.291126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:32.291297 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:32.291596 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:32.291601 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:32.291813 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 04:56:32.291935 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 04:56:32.292103 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260928 04:56:32.292856 13652 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38663
I20260928 04:56:32.292953 13652 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:32.293103 13652 heartbeater.cc:511] Master 127.0.98.62:38663 requested a full tablet report, sending...
W20260928 04:56:32.293745 13652 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38663 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 04:56:32.293812 13652 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:32.293860 13652 heartbeater.cc:511] Master 127.0.98.62:38663 requested a full tablet report, sending...
I20260928 04:56:32.294147 13652 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:32.294219 13652 heartbeater.cc:511] Master 127.0.98.62:38663 requested a full tablet report, sending...
W20260928 04:56:32.294510 13652 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 04:56:32.294750 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:32.284104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34794 (local address 127.0.98.62:38663)
0928 04:56:32.284459 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:32.284467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:32.284473 (+ 6us) server_negotiation.cc:407] Connection header received
0928 04:56:32.284491 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:32.284491 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:32.284500 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:32.284575 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:32.285185 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:32.285820 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:32.286834 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:32.287058 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:32.287151 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:32.289185 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:32.289186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:32.289187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:32.289194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:32.291026 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:32.291449 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:32.291451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:32.291451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:32.291532 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:32.291812 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:32.291813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:32.291813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:32.291946 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:32.292114 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:32.294567 (+ 2453us) server_negotiation.cc:299] Negotiation successful
0928 04:56:32.294659 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":91,"threads_started":1}
Sep 28 04:56:32 dist-test-slave-m39l krb5kdc[12190](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790571392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:32.876564 13384 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:32.876726 13384 heartbeater.cc:511] Master 127.0.98.62:38663 requested a full tablet report, sending...
I20260928 04:56:32.877125 13186 ts_manager.cc:194] Registered new tserver with Master: b6772fda83ae4260a6b240656b6aafe0 (127.0.98.1:33273)
I20260928 04:56:32.878110 13186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48785
I20260928 04:56:33.075907 13518 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:33.076102 13518 heartbeater.cc:511] Master 127.0.98.62:38663 requested a full tablet report, sending...
I20260928 04:56:33.076476 13186 ts_manager.cc:194] Registered new tserver with Master: 11a4f8c40a8d404c984c36453a8f0b6e (127.0.98.2:46541)
I20260928 04:56:33.077088 13186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42663
Sep 28 04:56:33 dist-test-slave-m39l krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:56:33.294673 13652 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:33.294853 13652 heartbeater.cc:511] Master 127.0.98.62:38663 requested a full tablet report, sending...
I20260928 04:56:33.295212 13186 ts_manager.cc:194] Registered new tserver with Master: 8cfe4f8bce6a49e7bc94e16b01c55b25 (127.0.98.3:46223)
I20260928 04:56:33.295650 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:56:33.295744 13186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46609
I20260928 04:56:33.305433 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:33.296950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34806 (local address 127.0.98.62:38663)
0928 04:56:33.297093 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:33.297095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:33.297103 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:56:33.297125 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:33.297126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:33.297138 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:33.297933 (+ 795us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:33.297980 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.298762 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.299811 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.300124 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.301871 (+ 1747us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:33.301930 (+ 59us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:33.301930 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:33.301931 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:33.301936 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:33.303849 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:33.304264 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:33.304265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:33.304265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:33.304328 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:33.304524 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:33.304524 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:33.304525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:33.304674 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:33.305194 (+ 520us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:33.305236 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 04:56:33.305323 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260928 04:56:33.308102 13186 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:56:33.310997 13186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:56:33.329152 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:33.323501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46223 (local address 127.0.0.1:56750)
0928 04:56:33.324071 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:33.324078 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:33.324146 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:33.324425 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:33.324428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:33.324430 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:33.324540 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:33.324542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.326326 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.326327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:33.327745 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:33.327748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.328807 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.328808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:33.328894 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:33.328910 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 04:56:33.328978 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 04:56:33.329038 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":55,"thread_start_us":74,"threads_started":1}
I20260928 04:56:33.329475 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:33.323091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52464 (local address 127.0.98.1:33273)
0928 04:56:33.324258 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:33.324263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:33.324275 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:56:33.324307 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:33.324309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:33.324323 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:33.324638 (+ 315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:33.325182 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.326629 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.328167 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.328894 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.329243 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:33.329341 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:33.329370 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 04:56:33.329407 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1131,"thread_start_us":54,"threads_started":1}
I20260928 04:56:33.329648 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:33.323269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.0.98.3:46223)
0928 04:56:33.324002 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:33.324007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:33.324146 (+ 139us) server_negotiation.cc:407] Connection header received
0928 04:56:33.324224 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:33.324226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:33.324244 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:33.324320 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:33.324625 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.326226 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.327852 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.328614 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.328687 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:33.328768 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:33.329516 (+ 748us) server_negotiation.cc:299] Negotiation successful
0928 04:56:33.329574 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":59,"threads_started":1}
I20260928 04:56:33.329166 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:33.322981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33273 (local address 127.0.0.1:52464)
0928 04:56:33.323863 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:33.323871 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:33.323915 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:33.324425 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:33.324428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:33.324433 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:33.324540 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:33.324542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.326723 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.326724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:33.327788 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:33.327790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.328990 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.328991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:33.329041 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:33.329044 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:56:33.329072 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 04:56:33.329106 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":2,"thread_start_us":85,"threads_started":1}
I20260928 04:56:33.330629 13576 tablet_service.cc:1467] Processing CreateTablet for tablet 8d64be228e084492aa06d1695296126f (DEFAULT_TABLE table=test-table [id=eb071f0689c043aab3de433996aab477]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:56:33.330633 13318 tablet_service.cc:1467] Processing CreateTablet for tablet 8d64be228e084492aa06d1695296126f (DEFAULT_TABLE table=test-table [id=eb071f0689c043aab3de433996aab477]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:56:33.331023 13576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d64be228e084492aa06d1695296126f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:33.331045 13318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d64be228e084492aa06d1695296126f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:33.335175 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:33.322722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46541 (local address 127.0.0.1:41504)
0928 04:56:33.323642 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:33.323694 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:33.323760 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:33.330932 (+ 7172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:33.330933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:33.330936 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:33.330989 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:33.330991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.332319 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.332321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:33.333410 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:33.333413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.334861 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.334862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:33.334932 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:33.334936 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:56:33.335028 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 04:56:33.335079 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"thread_start_us":83,"threads_started":1}
I20260928 04:56:33.335175 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:33.322801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.0.98.2:46541)
0928 04:56:33.330371 (+ 7570us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:33.330376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:33.330389 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:56:33.330426 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:33.330428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:33.330445 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:33.330605 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:33.331148 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.332213 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.333529 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.334684 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.334750 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:33.334827 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:33.335048 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 04:56:33.335097 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7529,"thread_start_us":61,"threads_started":1}
I20260928 04:56:33.336084 13452 tablet_service.cc:1467] Processing CreateTablet for tablet 8d64be228e084492aa06d1695296126f (DEFAULT_TABLE table=test-table [id=eb071f0689c043aab3de433996aab477]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:56:33.336418 13452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d64be228e084492aa06d1695296126f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:33.335686 13687 tablet_bootstrap.cc:492] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0: Bootstrap starting.
I20260928 04:56:33.339509 13687 tablet_bootstrap.cc:654] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:33.339761 13686 tablet_bootstrap.cc:492] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25: Bootstrap starting.
I20260928 04:56:33.340045 13688 tablet_bootstrap.cc:492] T 8d64be228e084492aa06d1695296126f P 11a4f8c40a8d404c984c36453a8f0b6e: Bootstrap starting.
I20260928 04:56:33.340204 13687 log.cc:824] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:33.340978 13688 tablet_bootstrap.cc:654] T 8d64be228e084492aa06d1695296126f P 11a4f8c40a8d404c984c36453a8f0b6e: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:33.341012 13686 tablet_bootstrap.cc:654] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:33.341362 13686 log.cc:824] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:33.341362 13688 log.cc:824] T 8d64be228e084492aa06d1695296126f P 11a4f8c40a8d404c984c36453a8f0b6e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:33.342167 13688 tablet_bootstrap.cc:492] T 8d64be228e084492aa06d1695296126f P 11a4f8c40a8d404c984c36453a8f0b6e: No bootstrap required, opened a new log
I20260928 04:56:33.342259 13688 ts_tablet_manager.cc:1397] T 8d64be228e084492aa06d1695296126f P 11a4f8c40a8d404c984c36453a8f0b6e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:56:33.342974 13686 tablet_bootstrap.cc:492] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25: No bootstrap required, opened a new log
I20260928 04:56:33.343150 13686 ts_tablet_manager.cc:1397] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 04:56:33.345052 13688 raft_consensus.cc:348] T 8d64be228e084492aa06d1695296126f P 11a4f8c40a8d404c984c36453a8f0b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cfe4f8bce6a49e7bc94e16b01c55b25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46223 } } peers { permanent_uuid: "11a4f8c40a8d404c984c36453a8f0b6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46541 } } peers { permanent_uuid: "b6772fda83ae4260a6b240656b6aafe0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33273 } }
I20260928 04:56:33.345748 13688 raft_consensus.cc:374] T 8d64be228e084492aa06d1695296126f P 11a4f8c40a8d404c984c36453a8f0b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:33.345855 13688 raft_consensus.cc:729] T 8d64be228e084492aa06d1695296126f P 11a4f8c40a8d404c984c36453a8f0b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11a4f8c40a8d404c984c36453a8f0b6e, State: Initialized, Role: FOLLOWER
I20260928 04:56:33.345861 13686 raft_consensus.cc:348] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cfe4f8bce6a49e7bc94e16b01c55b25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46223 } } peers { permanent_uuid: "11a4f8c40a8d404c984c36453a8f0b6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46541 } } peers { permanent_uuid: "b6772fda83ae4260a6b240656b6aafe0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33273 } }
I20260928 04:56:33.346022 13686 raft_consensus.cc:374] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:33.346063 13686 raft_consensus.cc:729] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cfe4f8bce6a49e7bc94e16b01c55b25, State: Initialized, Role: FOLLOWER
I20260928 04:56:33.346192 13686 consensus_queue.cc:260] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cfe4f8bce6a49e7bc94e16b01c55b25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46223 } } peers { permanent_uuid: "11a4f8c40a8d404c984c36453a8f0b6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46541 } } peers { permanent_uuid: "b6772fda83ae4260a6b240656b6aafe0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33273 } }
I20260928 04:56:33.346218 13688 consensus_queue.cc:260] T 8d64be228e084492aa06d1695296126f P 11a4f8c40a8d404c984c36453a8f0b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cfe4f8bce6a49e7bc94e16b01c55b25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46223 } } peers { permanent_uuid: "11a4f8c40a8d404c984c36453a8f0b6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46541 } } peers { permanent_uuid: "b6772fda83ae4260a6b240656b6aafe0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33273 } }
I20260928 04:56:33.346599 13686 ts_tablet_manager.cc:1428] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:56:33.346726 13688 ts_tablet_manager.cc:1428] T 8d64be228e084492aa06d1695296126f P 11a4f8c40a8d404c984c36453a8f0b6e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 04:56:33.346791 13652 heartbeater.cc:503] Master 127.0.98.62:38663 was elected leader, sending a full tablet report...
I20260928 04:56:33.346868 13518 heartbeater.cc:503] Master 127.0.98.62:38663 was elected leader, sending a full tablet report...
I20260928 04:56:33.347669 13687 tablet_bootstrap.cc:492] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0: No bootstrap required, opened a new log
I20260928 04:56:33.347749 13687 ts_tablet_manager.cc:1397] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260928 04:56:33.349746 13687 raft_consensus.cc:348] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cfe4f8bce6a49e7bc94e16b01c55b25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46223 } } peers { permanent_uuid: "11a4f8c40a8d404c984c36453a8f0b6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46541 } } peers { permanent_uuid: "b6772fda83ae4260a6b240656b6aafe0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33273 } }
I20260928 04:56:33.349869 13687 raft_consensus.cc:374] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:33.349889 13687 raft_consensus.cc:729] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6772fda83ae4260a6b240656b6aafe0, State: Initialized, Role: FOLLOWER
I20260928 04:56:33.350013 13687 consensus_queue.cc:260] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cfe4f8bce6a49e7bc94e16b01c55b25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46223 } } peers { permanent_uuid: "11a4f8c40a8d404c984c36453a8f0b6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46541 } } peers { permanent_uuid: "b6772fda83ae4260a6b240656b6aafe0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33273 } }
I20260928 04:56:33.350281 13687 ts_tablet_manager.cc:1428] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:56:33.350312 13384 heartbeater.cc:503] Master 127.0.98.62:38663 was elected leader, sending a full tablet report...
W20260928 04:56:33.357599 13385 tablet.cc:2369] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:56:33.411386 13694 raft_consensus.cc:482] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:56:33.411556 13694 raft_consensus.cc:504] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cfe4f8bce6a49e7bc94e16b01c55b25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46223 } } peers { permanent_uuid: "11a4f8c40a8d404c984c36453a8f0b6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46541 } } peers { permanent_uuid: "b6772fda83ae4260a6b240656b6aafe0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33273 } }
I20260928 04:56:33.411959 13694 leader_election.cc:288] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cfe4f8bce6a49e7bc94e16b01c55b25 (127.0.98.3:46223), 11a4f8c40a8d404c984c36453a8f0b6e (127.0.98.2:46541)
I20260928 04:56:33.416950 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:33.412395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40907 (local address 127.0.98.3:46223)
0928 04:56:33.412444 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:33.412447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:33.412584 (+ 137us) server_negotiation.cc:407] Connection header received
0928 04:56:33.412702 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:33.412703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:33.412715 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:33.412747 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:33.413048 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.413817 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.415898 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.416492 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.416548 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:33.416559 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:33.416823 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 04:56:33.416872 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:56:33.416950 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:33.412396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52029 (local address 127.0.98.2:46541)
0928 04:56:33.412444 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:33.412447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:33.412584 (+ 137us) server_negotiation.cc:407] Connection header received
0928 04:56:33.412701 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:33.412702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:33.412715 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:33.412747 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:33.413031 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.413801 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.415898 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.416512 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.416552 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:33.416560 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:33.416822 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 04:56:33.416873 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:56:33.417253 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:33.412141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46541 (local address 127.0.98.1:52029)
0928 04:56:33.412528 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:33.412538 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:33.412614 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:33.412819 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:33.412820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:33.412822 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:33.412942 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:33.412945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.413918 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.413919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:33.415776 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:33.415780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.416644 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.416645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:33.416732 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:33.416735 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:56:33.417086 (+ 351us) client_negotiation.cc:240] Negotiation successful
0928 04:56:33.417164 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":28,"thread_start_us":107,"threads_started":1}
I20260928 04:56:33.417610 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:33.412131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46223 (local address 127.0.98.1:40907)
0928 04:56:33.412511 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:33.412523 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:33.412620 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:33.412819 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:33.412820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:33.412823 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:33.412969 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:33.412970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.413918 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.413918 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:33.415776 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:33.415780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.416660 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.416661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:33.416725 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:33.416730 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:56:33.417487 (+ 757us) client_negotiation.cc:240] Negotiation successful
0928 04:56:33.417533 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":72,"thread_start_us":114,"threads_started":1}
I20260928 04:56:33.417591 13472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d64be228e084492aa06d1695296126f" candidate_uuid: "b6772fda83ae4260a6b240656b6aafe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11a4f8c40a8d404c984c36453a8f0b6e" is_pre_election: true
I20260928 04:56:33.417769 13472 raft_consensus.cc:2454] T 8d64be228e084492aa06d1695296126f P 11a4f8c40a8d404c984c36453a8f0b6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6772fda83ae4260a6b240656b6aafe0 in term 0.
I20260928 04:56:33.417995 13602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d64be228e084492aa06d1695296126f" candidate_uuid: "b6772fda83ae4260a6b240656b6aafe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cfe4f8bce6a49e7bc94e16b01c55b25" is_pre_election: true
I20260928 04:56:33.418099 13272 leader_election.cc:302] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11a4f8c40a8d404c984c36453a8f0b6e, b6772fda83ae4260a6b240656b6aafe0; no voters:
I20260928 04:56:33.418170 13602 raft_consensus.cc:2454] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6772fda83ae4260a6b240656b6aafe0 in term 0.
I20260928 04:56:33.418232 13694 raft_consensus.cc:2781] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:56:33.418275 13694 raft_consensus.cc:482] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:56:33.418306 13694 raft_consensus.cc:3037] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:33.419096 13694 raft_consensus.cc:504] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cfe4f8bce6a49e7bc94e16b01c55b25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46223 } } peers { permanent_uuid: "11a4f8c40a8d404c984c36453a8f0b6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46541 } } peers { permanent_uuid: "b6772fda83ae4260a6b240656b6aafe0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33273 } }
I20260928 04:56:33.419255 13694 leader_election.cc:288] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [CANDIDATE]: Term 1 election: Requested vote from peers 8cfe4f8bce6a49e7bc94e16b01c55b25 (127.0.98.3:46223), 11a4f8c40a8d404c984c36453a8f0b6e (127.0.98.2:46541)
I20260928 04:56:33.419405 13602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d64be228e084492aa06d1695296126f" candidate_uuid: "b6772fda83ae4260a6b240656b6aafe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cfe4f8bce6a49e7bc94e16b01c55b25"
I20260928 04:56:33.419473 13602 raft_consensus.cc:3037] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:33.419467 13472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d64be228e084492aa06d1695296126f" candidate_uuid: "b6772fda83ae4260a6b240656b6aafe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11a4f8c40a8d404c984c36453a8f0b6e"
I20260928 04:56:33.419554 13472 raft_consensus.cc:3037] T 8d64be228e084492aa06d1695296126f P 11a4f8c40a8d404c984c36453a8f0b6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:33.420285 13602 raft_consensus.cc:2454] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6772fda83ae4260a6b240656b6aafe0 in term 1.
I20260928 04:56:33.420512 13273 leader_election.cc:302] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cfe4f8bce6a49e7bc94e16b01c55b25, b6772fda83ae4260a6b240656b6aafe0; no voters:
I20260928 04:56:33.420516 13472 raft_consensus.cc:2454] T 8d64be228e084492aa06d1695296126f P 11a4f8c40a8d404c984c36453a8f0b6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6772fda83ae4260a6b240656b6aafe0 in term 1.
I20260928 04:56:33.420607 13694 raft_consensus.cc:2781] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:56:33.420780 13694 raft_consensus.cc:686] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [term 1 LEADER]: Becoming Leader. State: Replica: b6772fda83ae4260a6b240656b6aafe0, State: Running, Role: LEADER
I20260928 04:56:33.420888 13694 consensus_queue.cc:237] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cfe4f8bce6a49e7bc94e16b01c55b25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46223 } } peers { permanent_uuid: "11a4f8c40a8d404c984c36453a8f0b6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46541 } } peers { permanent_uuid: "b6772fda83ae4260a6b240656b6aafe0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33273 } }
I20260928 04:56:33.421756 13186 catalog_manager.cc:5495] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6772fda83ae4260a6b240656b6aafe0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b6772fda83ae4260a6b240656b6aafe0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cfe4f8bce6a49e7bc94e16b01c55b25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11a4f8c40a8d404c984c36453a8f0b6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6772fda83ae4260a6b240656b6aafe0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33273 } health_report { overall_health: HEALTHY } } }
I20260928 04:56:33.445377 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:33.441954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52472 (local address 127.0.98.1:33273)
0928 04:56:33.442075 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:33.442077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:33.442086 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:56:33.442130 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:33.442131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:33.442156 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:33.442229 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:56:33.442430 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.443223 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.444391 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:33.444700 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:33.444772 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:33.444973 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:56:33.445124 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:56:33.445211 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:33.445264 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 04:56:33.445290 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 04:56:33.447678 13472 raft_consensus.cc:1261] T 8d64be228e084492aa06d1695296126f P 11a4f8c40a8d404c984c36453a8f0b6e [term 1 FOLLOWER]: Refusing update from remote peer b6772fda83ae4260a6b240656b6aafe0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:56:33.447705 13602 raft_consensus.cc:1261] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25 [term 1 FOLLOWER]: Refusing update from remote peer b6772fda83ae4260a6b240656b6aafe0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:56:33.448065 13694 consensus_queue.cc:1035] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cfe4f8bce6a49e7bc94e16b01c55b25" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:56:33.448238 13694 consensus_queue.cc:1035] T 8d64be228e084492aa06d1695296126f P b6772fda83ae4260a6b240656b6aafe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11a4f8c40a8d404c984c36453a8f0b6e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:56:33.451920 13700 mvcc.cc:205] Tried to move back new op lower bound from 7334180427558760448 to 7334180427452674048. Current Snapshot: MvccSnapshot[applied={T|T < 7334180427558760448}]
I20260928 04:56:33.452157 13701 mvcc.cc:205] Tried to move back new op lower bound from 7334180427558760448 to 7334180427452674048. Current Snapshot: MvccSnapshot[applied={T|T < 7334180427558760448}]
I20260928 04:56:33.454314 13702 mvcc.cc:205] Tried to move back new op lower bound from 7334180427558760448 to 7334180427452674048. Current Snapshot: MvccSnapshot[applied={T|T < 7334180427558760448 or (T in {7334180427558760448})}]
I20260928 04:56:33.457105 13186 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:56:33.457234 13186 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:56:33.458041 13186 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c32927b9dd3e4d54bba766e8c6f30677: Sending DeleteTablet for 3 replicas of tablet 8d64be228e084492aa06d1695296126f
I20260928 04:56:33.458767 13452 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d64be228e084492aa06d1695296126f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:56:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41504
I20260928 04:56:33.459025 13576 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d64be228e084492aa06d1695296126f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:56:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56750
I20260928 04:56:33.459035 13318 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d64be228e084492aa06d1695296126f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:56:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52464
I20260928 04:56:33.459463 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 13224
I20260928 04:56:33.462666 13711 tablet_replica.cc:330] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25: stopping tablet replica
I20260928 04:56:33.462841 13711 raft_consensus.cc:2229] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:56:33.463063 13711 raft_consensus.cc:2258] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:56:33.463599 13711 ts_tablet_manager.cc:1905] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 04:56:33.465435 13170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33273 (error 108)
I20260928 04:56:33.465698 13711 ts_tablet_manager.cc:1918] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:56:33.465857 13711 log.cc:1196] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/8d64be228e084492aa06d1695296126f
I20260928 04:56:33.466089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 13388
I20260928 04:56:33.468026 13711 ts_tablet_manager.cc:1939] T 8d64be228e084492aa06d1695296126f P 8cfe4f8bce6a49e7bc94e16b01c55b25: Deleting consensus metadata
I20260928 04:56:33.468626 13172 catalog_manager.cc:4841] TS 8cfe4f8bce6a49e7bc94e16b01c55b25 (127.0.98.3:46223): tablet 8d64be228e084492aa06d1695296126f (table test-table [id=eb071f0689c043aab3de433996aab477]) successfully deleted
I20260928 04:56:33.469025 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:33.468895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33273 (local address 127.0.0.1:52482)
0928 04:56:33.468937 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:33.468984 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 04:56:33.469177 13170 catalog_manager.cc:4584] TS b6772fda83ae4260a6b240656b6aafe0 (127.0.98.1:33273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d64be228e084492aa06d1695296126f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33273: connect: Connection refused (error 111)
I20260928 04:56:33.474773 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 13521
I20260928 04:56:33.474864 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:33.474751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46541 (local address 127.0.0.1:41506)
0928 04:56:33.474803 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:33.474826 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 04:56:33.480969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 13153
I20260928 04:56:33.487905 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 04:56:33.833055 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 04:56:33.833181 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 04:56:34.178092 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 04:56:34.178 UTC [12220] LOG: received smart shutdown request
2026-09-28 04:56:34.182 UTC [12220] LOG: background worker "logical replication launcher" (PID 12229) exited with exit code 1
2026-09-28 04:56:34.182 UTC [12224] LOG: shutting down
2026-09-28 04:56:34.403 UTC [12220] LOG: database system is shut down
2026-09-28T04:56:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52410 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:56:34 dist-test-slave-m39l krb5kdc[13722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:56:34 dist-test-slave-m39l krb5kdc[13722](info): set up 2 sockets
Sep 28 04:56:34 dist-test-slave-m39l krb5kdc[13722](info): commencing operation
krb5kdc: starting...
W20260928 04:56:35.582518 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:56:35 dist-test-slave-m39l krb5kdc[13722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:56:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:56:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:56:35.710069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33783
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33783,127.0.98.61:38193,127.0.98.60:42983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:56:35.793200 13735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:35.793466 13735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:35.793519 13735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:35.794709 13735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:56:35.794780 13735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:35.794806 13735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:35.794832 13735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:56:35.794862 13735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:56:35.796399 13735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33783,127.0.98.61:38193,127.0.98.60:42983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:35.796694 13735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:35.796928 13735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:35.802021 13743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:35.802034 13740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:35.802211 13735 server_base.cc:1034] running on GCE node
W20260928 04:56:35.802119 13741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:35.802685 13735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:35.803045 13735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:35.804225 13735 hybrid_clock.cc:648] HybridClock initialized: now 1790571395804206 us; error 31 us; skew 500 ppm
Sep 28 04:56:35 dist-test-slave-m39l krb5kdc[13722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:35.812160 13735 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:56:35.812691 13735 webserver.cc:460] Webserver started at http://127.0.98.62:35495/ using document root <none> and password file <none>
I20260928 04:56:35.812927 13735 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:35.812995 13735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:35.813148 13735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:56:35.814254 13735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "63122aabe8d1411a895ee5576aab4b15"
format_stamp: "Formatted at 2026-09-28 04:56:35 on dist-test-slave-m39l"
server_key: "9333799f54146aa78659e51438f1b6d3"
server_key_iv: "b046f50eb7137f925f85246de88b02d9"
server_key_version: "encryptionkey@0"
I20260928 04:56:35.814667 13735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "63122aabe8d1411a895ee5576aab4b15"
format_stamp: "Formatted at 2026-09-28 04:56:35 on dist-test-slave-m39l"
server_key: "9333799f54146aa78659e51438f1b6d3"
server_key_iv: "b046f50eb7137f925f85246de88b02d9"
server_key_version: "encryptionkey@0"
I20260928 04:56:35.816095 13735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:56:35.817050 13750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:35.817267 13735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:35.817378 13735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "63122aabe8d1411a895ee5576aab4b15"
format_stamp: "Formatted at 2026-09-28 04:56:35 on dist-test-slave-m39l"
server_key: "9333799f54146aa78659e51438f1b6d3"
server_key_iv: "b046f50eb7137f925f85246de88b02d9"
server_key_version: "encryptionkey@0"
I20260928 04:56:35.817487 13735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:35.836778 13735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:35.839841 13735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:35.840018 13735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:35.844434 13735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33783
I20260928 04:56:35.844489 13802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33783 every 8 connection(s)
I20260928 04:56:35.844907 13735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:56:35.845626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 13735
I20260928 04:56:35.845759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:56:35.845983 392 external_mini_cluster.cc:1428] Setting key b91953b57e3e408dac73cf3e12db9cf9
I20260928 04:56:35.846038 13803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:35.848310 13803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 28 04:56:35 dist-test-slave-m39l krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:35.862835 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:35.849313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.0.98.62:33783)
0928 04:56:35.849485 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:35.849496 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:35.849752 (+ 256us) server_negotiation.cc:407] Connection header received
0928 04:56:35.850546 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:35.850548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:35.850616 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:35.850699 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:35.851537 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:35.852374 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:35.853777 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:35.854108 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:35.854194 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:35.858839 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:35.858849 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:35.858850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:35.858859 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:35.860804 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:35.861347 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:35.861350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:35.861351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:35.861448 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:35.861763 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:35.861764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:35.861765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:35.862049 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:35.862128 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:35.862404 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 04:56:35.862595 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260928 04:56:35.862922 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:35.849272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33783 (local address 127.0.0.1:59158)
0928 04:56:35.849571 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:35.849621 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:35.850335 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:35.850753 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:35.850757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:35.851162 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:35.851430 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:35.851436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:35.852479 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:35.852480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:35.853682 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:35.853685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:35.854199 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:35.854200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:35.854286 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:35.854962 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:35.854971 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:35.858735 (+ 3764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:35.860907 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:35.860915 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:35.860917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:35.861222 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:35.861530 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:35.861531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:35.861531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:35.861665 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:35.862132 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:35.862140 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:35.862229 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 04:56:35.862341 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 04:56:35.862706 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":159,"thread_start_us":101,"threads_started":1}
I20260928 04:56:35.864308 13803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:56:35.864729 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:35.864599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38193 (local address 127.0.0.1:38242)
0928 04:56:35.864653 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:35.864702 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 04:56:35.864952 13752 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38193: connect: Connection refused (error 111)
I20260928 04:56:35.865598 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:35.865477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38193 (local address 127.0.0.1:38246)
0928 04:56:35.865534 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:35.865573 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 04:56:35.865758 13803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38193: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38193: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 04:56:35.878118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38193
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33783,127.0.98.61:38193,127.0.98.60:42983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:56:35.915024 13803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } attempt: 1
I20260928 04:56:35.915545 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:35.915404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38193 (local address 127.0.0.1:38252)
0928 04:56:35.915470 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:35.915513 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 04:56:35.915915 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:35.915824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38193 (local address 127.0.0.1:38262)
0928 04:56:35.915865 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:35.915896 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 04:56:35.915997 13803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38193: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38193: connect: Connection refused (error 111)
W20260928 04:56:35.962471 13810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:35.962743 13810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:35.962770 13810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:35.964068 13810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:56:35.964128 13810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:35.964151 13810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:35.964210 13810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:56:35.964233 13810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:56:35.965878 13810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33783,127.0.98.61:38193,127.0.98.60:42983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:35.966166 13810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:35.966411 13810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:35.971374 13818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:35.971375 13815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:35.971375 13816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:35.971854 13810 server_base.cc:1034] running on GCE node
I20260928 04:56:35.972115 13810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:35.972448 13810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:35.973654 13810 hybrid_clock.cc:648] HybridClock initialized: now 1790571395973639 us; error 31 us; skew 500 ppm
Sep 28 04:56:35 dist-test-slave-m39l krb5kdc[13722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:35.979135 13810 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 04:56:35.979736 13810 webserver.cc:460] Webserver started at http://127.0.98.61:45329/ using document root <none> and password file <none>
I20260928 04:56:35.980005 13810 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:35.980072 13810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:35.980227 13810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:56:35.981387 13810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-1/data/instance:
uuid: "14290cc0eaa5495b86d709bda88c3fa4"
format_stamp: "Formatted at 2026-09-28 04:56:35 on dist-test-slave-m39l"
server_key: "10e7c9a151ca2515cd88b51d319604b3"
server_key_iv: "fa6735a11f7d2733307006a9a084a731"
server_key_version: "encryptionkey@0"
I20260928 04:56:35.981768 13810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-1/wal/instance:
uuid: "14290cc0eaa5495b86d709bda88c3fa4"
format_stamp: "Formatted at 2026-09-28 04:56:35 on dist-test-slave-m39l"
server_key: "10e7c9a151ca2515cd88b51d319604b3"
server_key_iv: "fa6735a11f7d2733307006a9a084a731"
server_key_version: "encryptionkey@0"
I20260928 04:56:35.983347 13810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:56:35.984124 13803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } attempt: 2
I20260928 04:56:35.984313 13825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:35.984519 13810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:35.984603 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:35.984458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38193 (local address 127.0.0.1:38278)
0928 04:56:35.984515 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:35.984552 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 04:56:35.984615 13810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-1/wal
uuid: "14290cc0eaa5495b86d709bda88c3fa4"
format_stamp: "Formatted at 2026-09-28 04:56:35 on dist-test-slave-m39l"
server_key: "10e7c9a151ca2515cd88b51d319604b3"
server_key_iv: "fa6735a11f7d2733307006a9a084a731"
server_key_version: "encryptionkey@0"
I20260928 04:56:35.984694 13810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:35.985073 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:35.984974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38193 (local address 127.0.0.1:38282)
0928 04:56:35.985017 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:35.985043 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 04:56:35.985163 13803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38193: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38193: connect: Connection refused (error 111)
I20260928 04:56:35.994443 13810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:35.996821 13810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:35.997035 13810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:36.001775 13810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38193
I20260928 04:56:36.001808 13877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38193 every 8 connection(s)
I20260928 04:56:36.002211 13810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-1/data/info.pb
I20260928 04:56:36.003271 13878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:36.003320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 13810
I20260928 04:56:36.003422 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-1/wal/instance
I20260928 04:56:36.003641 392 external_mini_cluster.cc:1428] Setting key 3acde38b7be00f3fe7a29f371bbc2e99
I20260928 04:56:36.005658 13878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 04:56:36 dist-test-slave-m39l krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:36.019671 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.006440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59174 (local address 127.0.98.62:33783)
0928 04:56:36.006539 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:36.006542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:36.006728 (+ 186us) server_negotiation.cc:407] Connection header received
0928 04:56:36.007185 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:36.007187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:36.007203 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:36.007287 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:36.007942 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.008806 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.009823 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.010175 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.010262 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.016145 (+ 5883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:36.016148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:36.016148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:36.016160 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:36.017981 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.018468 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.018470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.018470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.018565 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.018868 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.018869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.018869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.019045 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:36.019121 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:36.019455 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 04:56:36.019559 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 04:56:36.019948 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.006392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33783 (local address 127.0.0.1:59174)
0928 04:56:36.006586 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:36.006639 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:36.007085 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:36.007367 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:36.007371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:36.007638 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:36.007840 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:36.007844 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.008912 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.008913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:36.009732 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:36.009734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.010264 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.010265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:36.010372 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.010961 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:36.010970 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:36.015992 (+ 5022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:36.018077 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:36.018087 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:36.018089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:36.018383 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:36.018663 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:36.018664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:36.018664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:36.018797 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:36.019124 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:36.019132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:36.019310 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 04:56:36.019410 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 04:56:36.019623 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":106,"threads_started":1}
I20260928 04:56:36.020601 13878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:56:36 dist-test-slave-m39l krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:56:36.030865 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.020949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38193 (local address 127.0.0.1:38296)
0928 04:56:36.021004 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:36.021017 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:36.021069 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:36.021460 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:36.021461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:36.021619 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:36.021678 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:36.021680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.022915 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.022917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:36.023491 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:36.023493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.024026 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.024027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:36.024100 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.024524 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:36.024530 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:36.027282 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:36.029526 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:36.029527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:36.029528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:36.029708 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:36.029946 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:36.029947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:36.029947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:36.030012 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:36.030419 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:36.030420 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:36.030465 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 04:56:36.030621 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 04:56:36.030741 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 04:56:36.031106 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.021014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.0.98.61:38193)
0928 04:56:36.021301 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:36.021306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:36.021327 (+ 21us) server_negotiation.cc:407] Connection header received
0928 04:56:36.021350 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:36.021352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:36.021367 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:36.021460 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:36.021767 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.022819 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.023591 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.023931 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.024026 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.027407 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:36.027417 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:36.027417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:36.027426 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:36.029427 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.029811 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.029814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.029815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.029877 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.030098 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.030099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.030099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.030295 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:36.030425 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:36.030656 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 04:56:36.030869 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":236,"thread_start_us":191,"threads_started":1}
I20260928 04:56:36.032223 13878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:56:36.032627 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.032492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42983 (local address 127.0.0.1:55210)
0928 04:56:36.032544 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:36.032593 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 04:56:36.032750 13828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42983: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 04:56:36.033242 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.033116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42983 (local address 127.0.0.1:55214)
0928 04:56:36.033177 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:36.033213 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 04:56:36.033409 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42983: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42983: connect: Connection refused (error 111)
I20260928 04:56:36.033833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42983
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33783,127.0.98.61:38193,127.0.98.60:42983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:56:36.076350 13803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } attempt: 3
Sep 28 04:56:36 dist-test-slave-m39l krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:56:36.082625 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } attempt: 1
I20260928 04:56:36.083058 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.082938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42983 (local address 127.0.0.1:55228)
0928 04:56:36.082979 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:36.083015 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 04:56:36.083415 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.083308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42983 (local address 127.0.0.1:55238)
0928 04:56:36.083356 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:36.083384 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 04:56:36.083576 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42983: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42983: connect: Connection refused (error 111)
I20260928 04:56:36.085007 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.076828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38193 (local address 127.0.0.1:38304)
0928 04:56:36.076920 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:36.076934 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:36.076985 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:36.077188 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:36.077189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:36.077368 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:36.077438 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:36.077440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.078426 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.078427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:36.079111 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:36.079114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.079630 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.079632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:36.079724 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.080307 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:36.080312 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:36.081720 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:36.083951 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:36.083952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:36.083953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:36.084133 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:36.084358 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:36.084359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:36.084360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:36.084433 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:36.084723 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:36.084723 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:36.084771 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 04:56:36.084833 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 04:56:36.084936 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 04:56:36.085050 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.076916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38304 (local address 127.0.98.61:38193)
0928 04:56:36.076982 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:36.076985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:36.076997 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:56:36.077079 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:36.077081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:36.077093 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:36.077173 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:36.077520 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.078326 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.079204 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.079522 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.079601 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.081822 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:36.081824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:36.081824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:36.081832 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:36.083829 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.084200 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.084201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.084201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.084264 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.084506 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.084506 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.084507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.084650 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:36.084717 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:36.084872 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 04:56:36.084959 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:56:36.085445 13803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:56:36.085875 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.085751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42983 (local address 127.0.0.1:55248)
0928 04:56:36.085808 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:36.085840 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 04:56:36.086233 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.086149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42983 (local address 127.0.0.1:55262)
0928 04:56:36.086193 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:36.086218 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 04:56:36.086341 13803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42983: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42983: connect: Connection refused (error 111)
I20260928 04:56:36.117539 13803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } attempt: 1
I20260928 04:56:36.118233 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.118081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42983 (local address 127.0.0.1:55272)
0928 04:56:36.118156 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:36.118204 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 04:56:36.118769 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.118633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42983 (local address 127.0.0.1:55274)
0928 04:56:36.118683 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:36.118735 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 04:56:36.118878 13803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42983: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42983: connect: Connection refused (error 111)
W20260928 04:56:36.124089 13885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:36.124334 13885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:36.124373 13885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:36.125620 13885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:56:36.125697 13885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:36.125723 13885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:36.125754 13885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:56:36.125782 13885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:56:36.127440 13885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33783,127.0.98.61:38193,127.0.98.60:42983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:36.127729 13885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:36.127959 13885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:36.133373 13893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:36.133488 13885 server_base.cc:1034] running on GCE node
W20260928 04:56:36.133391 13890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:36.133390 13891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:36.133908 13885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:36.134312 13885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:36.135514 13885 hybrid_clock.cc:648] HybridClock initialized: now 1790571396135479 us; error 43 us; skew 500 ppm
Sep 28 04:56:36 dist-test-slave-m39l krb5kdc[13722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:36.138484 13885 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 04:56:36.138988 13885 webserver.cc:460] Webserver started at http://127.0.98.60:45327/ using document root <none> and password file <none>
I20260928 04:56:36.139252 13885 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:36.139303 13885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:36.139464 13885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:56:36.140702 13885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-2/data/instance:
uuid: "965cd4de176d483da7fcc0a5302526d6"
format_stamp: "Formatted at 2026-09-28 04:56:36 on dist-test-slave-m39l"
server_key: "8b02b19aee372e321d020a50a86366d7"
server_key_iv: "0bc5b1004c084d33e3437a382f529a8d"
server_key_version: "encryptionkey@0"
I20260928 04:56:36.141103 13885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-2/wal/instance:
uuid: "965cd4de176d483da7fcc0a5302526d6"
format_stamp: "Formatted at 2026-09-28 04:56:36 on dist-test-slave-m39l"
server_key: "8b02b19aee372e321d020a50a86366d7"
server_key_iv: "0bc5b1004c084d33e3437a382f529a8d"
server_key_version: "encryptionkey@0"
I20260928 04:56:36.142701 13885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:56:36.143697 13900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:36.143887 13885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:36.143972 13885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-2/wal
uuid: "965cd4de176d483da7fcc0a5302526d6"
format_stamp: "Formatted at 2026-09-28 04:56:36 on dist-test-slave-m39l"
server_key: "8b02b19aee372e321d020a50a86366d7"
server_key_iv: "0bc5b1004c084d33e3437a382f529a8d"
server_key_version: "encryptionkey@0"
I20260928 04:56:36.144028 13885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:36.151734 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } attempt: 2
I20260928 04:56:36.152274 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.152121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42983 (local address 127.0.0.1:55276)
0928 04:56:36.152185 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:36.152231 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 04:56:36.152691 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.152584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42983 (local address 127.0.0.1:55282)
0928 04:56:36.152632 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:36.152661 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 04:56:36.152791 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42983: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42983: connect: Connection refused (error 111)
I20260928 04:56:36.154915 13885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:36.157009 13885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:36.157169 13885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:36.161355 13885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42983
I20260928 04:56:36.161393 13952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42983 every 8 connection(s)
I20260928 04:56:36.161784 13885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-2/data/info.pb
I20260928 04:56:36.162833 13953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:36.164568 13953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:56:36.169721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 13885
I20260928 04:56:36.169821 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790571208412331-392-0/minicluster-data/master-2/wal/instance
I20260928 04:56:36.170028 392 external_mini_cluster.cc:1428] Setting key a1289bb0c41d04183728207a82494cfd
Sep 28 04:56:36 dist-test-slave-m39l krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:36.174780 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.165396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59184 (local address 127.0.98.62:33783)
0928 04:56:36.165469 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:36.165471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:36.165764 (+ 293us) server_negotiation.cc:407] Connection header received
0928 04:56:36.166221 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:36.166222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:36.166233 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:36.166296 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:36.167033 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.167619 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.168679 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.168865 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.168931 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.171554 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:36.171556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:36.171556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:36.171564 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:36.173112 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.173567 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.173568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.173569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.173638 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.173950 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.173951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.173951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.174077 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:36.174134 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:36.174633 (+ 499us) server_negotiation.cc:299] Negotiation successful
0928 04:56:36.174708 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:56:36.174979 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.165371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33783 (local address 127.0.0.1:59184)
0928 04:56:36.165611 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:36.165657 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:36.166125 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:36.166374 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:36.166377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:36.166724 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:36.166940 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:36.166944 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.167720 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.167721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:36.168581 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:36.168584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.168944 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.168945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:36.169048 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.169661 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:36.169671 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:36.171447 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:36.173219 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:36.173227 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:36.173229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:36.173482 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:36.173711 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:36.173712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:36.173712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:36.173875 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:36.174148 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:36.174155 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:36.174351 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 04:56:36.174514 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 04:56:36.174692 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":126,"threads_started":1}
Sep 28 04:56:36 dist-test-slave-m39l krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:56:36.176271 13953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:56:36.178823 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.170459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55286 (local address 127.0.98.60:42983)
0928 04:56:36.170691 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:36.170696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:36.170751 (+ 55us) server_negotiation.cc:407] Connection header received
0928 04:56:36.170777 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:36.170779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:36.170804 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:36.170893 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:36.171531 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.172253 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.172988 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.173310 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.173390 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.175786 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:36.175794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:36.175794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:36.175800 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:36.177491 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.177878 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.177881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.177882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.177947 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.178159 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.178160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.178161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.178314 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:36.178534 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:36.178609 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 04:56:36.178696 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":152,"threads_started":1}
Sep 28 04:56:36 dist-test-slave-m39l krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:56:36.183194 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.176582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38193 (local address 127.0.0.1:38310)
0928 04:56:36.176630 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:36.176648 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:36.176689 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:36.176925 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:36.176926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:36.177091 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:36.177122 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:36.177123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.177735 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.177736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:36.178218 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:36.178220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.178613 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.178614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:36.178689 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.179107 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:36.179112 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:36.180609 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:36.182105 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:36.182106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:36.182106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:36.182331 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:36.182522 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:36.182522 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:36.182523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:36.182602 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:36.182893 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:36.182894 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:36.182947 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 04:56:36.183027 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 04:56:36.183125 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 04:56:36.183264 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.176652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.0.98.61:38193)
0928 04:56:36.176817 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:36.176819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:36.176826 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:56:36.176848 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:36.176849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:36.176858 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:36.176921 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:36.177195 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.177661 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.178300 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.178557 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.178618 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.180687 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:36.180688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:36.180688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:36.180695 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:36.182034 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.182401 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.182402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.182402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.182449 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.182671 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.182672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.182672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.182817 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:36.182881 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:36.183074 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 04:56:36.183159 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260928 04:56:36.183765 13953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:56:36 dist-test-slave-m39l krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 04:56:36 dist-test-slave-m39l krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:56:36.188072 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.179806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59196 (local address 127.0.98.62:33783)
0928 04:56:36.179855 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:36.179857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:36.179866 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:56:36.179889 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:36.179889 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:36.179899 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:36.179966 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:36.180497 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.181242 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.181992 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.182333 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.182412 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.185125 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:36.185127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:36.185127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:36.185136 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:36.186748 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.187099 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.187100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.187101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.187160 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.187344 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.187345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.187345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.187486 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:36.187794 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:36.187877 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 04:56:36.187972 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 04:56:36.191540 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.184054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42983 (local address 127.0.0.1:55298)
0928 04:56:36.184099 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:36.184109 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:36.184145 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:36.184286 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:36.184287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:36.184401 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:36.184431 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:36.184432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.185141 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.185142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:36.185770 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:36.185772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.186330 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.186331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:36.186370 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.186785 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:36.186790 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:36.188206 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:36.190370 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:36.190371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:36.190372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:36.190661 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:36.190854 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:36.190855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:36.190855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:36.190923 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:36.191132 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:36.191132 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:36.191207 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 04:56:36.191282 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 04:56:36.191377 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 04:56:36.191627 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.184081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55298 (local address 127.0.98.60:42983)
0928 04:56:36.184124 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:36.184126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:36.184144 (+ 18us) server_negotiation.cc:407] Connection header received
0928 04:56:36.184207 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:36.184208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:36.184219 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:36.184283 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:36.184496 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.185051 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.185861 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.186240 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.186312 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.188299 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:36.188300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:36.188300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:36.188307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:36.190238 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.190740 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.190741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.190742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.190792 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.190983 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.190983 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.190984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.191078 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:36.191125 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:36.191300 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 04:56:36.191458 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":22}
I20260928 04:56:36.193001 13953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6: Bootstrap starting.
Sep 28 04:56:36 dist-test-slave-m39l krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:56:36.194023 13803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } attempt: 2
I20260928 04:56:36.194607 13953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:36.194972 13953 log.cc:824] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:36.195874 13953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6: No bootstrap required, opened a new log
I20260928 04:56:36.198226 13953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } }
Sep 28 04:56:36 dist-test-slave-m39l krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:56:36.198405 13953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:36.198706 13953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965cd4de176d483da7fcc0a5302526d6, State: Initialized, Role: FOLLOWER
I20260928 04:56:36.198992 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.188755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38326 (local address 127.0.98.61:38193)
0928 04:56:36.188827 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:36.188829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:36.188837 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:56:36.188862 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:36.188863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:36.188873 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:36.188943 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:36.189621 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.190452 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.191477 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.191902 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.191981 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.194727 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:36.194729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:36.194729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:36.194737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:36.196945 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.197363 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.197365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.197365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.197420 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.197640 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.197641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.197641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.197788 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:36.197965 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:36.198805 (+ 840us) server_negotiation.cc:299] Negotiation successful
0928 04:56:36.198887 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 04:56:36.198982 13953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } }
I20260928 04:56:36.199354 13953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:56:36.199324 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } } }
I20260928 04:56:36.199537 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:56:36.200508 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:56:36.201872 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.194311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42983 (local address 127.0.0.1:55302)
0928 04:56:36.194371 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:36.194383 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:36.194423 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:36.194656 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:36.194657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:36.194824 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:36.194871 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:36.194873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.195753 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.195755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:36.196311 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:36.196314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.196813 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.196814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:36.196878 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.197300 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:36.197305 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:36.198848 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:36.200690 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:36.200692 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:36.200692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:36.200915 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:36.201173 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:36.201174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:36.201174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:36.201240 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:36.201548 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:36.201549 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:36.201595 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 04:56:36.201671 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 04:56:36.201773 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 04:56:36.201443 13970 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 04:56:36.201867 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.194369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55302 (local address 127.0.98.60:42983)
0928 04:56:36.194402 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:36.194404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:36.194412 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:56:36.194551 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:36.194551 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:36.194560 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:36.194628 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:36.194961 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.195667 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.196391 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.196719 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.196802 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.198940 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:36.198941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:36.198941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:36.198948 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:36.200551 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.201033 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.201035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.201035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.201091 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.201307 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.201308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.201309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.201456 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:36.201520 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:36.201712 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 04:56:36.201783 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 04:56:36.202601 13970 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:56:36.203985 13803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15: Bootstrap starting.
I20260928 04:56:36.204778 13803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:36.205109 13803 log.cc:824] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:36.205947 13803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15: No bootstrap required, opened a new log
I20260928 04:56:36.205480 13885 master_runner.cc:186] Error getting master registration for 127.0.98.62:33783: OK, instance_id { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" instance_seqno: 1790571395843136 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 04:56:36.208935 13885 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 04:56:36.209408 13803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } }
I20260928 04:56:36.209569 13803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:36.209625 13803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63122aabe8d1411a895ee5576aab4b15, State: Initialized, Role: FOLLOWER
I20260928 04:56:36.209795 13803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } }
I20260928 04:56:36.210363 13803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:56:36.210865 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.202751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59204 (local address 127.0.98.62:33783)
0928 04:56:36.202798 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:36.202800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:36.203254 (+ 454us) server_negotiation.cc:407] Connection header received
0928 04:56:36.203327 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:36.203328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:36.203337 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:36.203387 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:36.204032 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.204515 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.205625 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.205918 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.205956 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.207701 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:36.207703 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:36.207703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:36.207711 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:36.209324 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.209797 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.209798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.209799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.209875 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.210129 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.210130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.210131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.210282 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:36.210358 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:36.210668 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 04:56:36.210763 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:56:36.213291 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.202366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38340 (local address 127.0.98.61:38193)
0928 04:56:36.202405 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:36.202407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:36.203021 (+ 614us) server_negotiation.cc:407] Connection header received
0928 04:56:36.203131 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:36.203132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:36.203142 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:36.203198 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:36.203906 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.204661 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.205625 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.205915 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.205956 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.210270 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:36.210271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:36.210272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:36.210280 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:36.212011 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.212404 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.212405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.212405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.212463 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.212672 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.212673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.212674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.212808 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:36.212896 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:36.213110 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 04:56:36.213199 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:56:36.214478 13735 master_runner.cc:186] Error getting master registration for 127.0.98.61:38193: OK, instance_id { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" instance_seqno: 1790571396000445 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 04:56:36.217856 13735 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 04:56:36.219269 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.203062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55304 (local address 127.0.98.60:42983)
0928 04:56:36.203095 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:36.203096 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:36.211186 (+ 8090us) server_negotiation.cc:407] Connection header received
0928 04:56:36.211319 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:36.211320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:36.211333 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:36.211370 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:36.212023 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.212891 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.214037 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.214390 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.214440 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.216056 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:36.216057 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:36.216058 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:36.216067 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:36.217938 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.218303 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.218304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.218305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.218370 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.218704 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.218706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.218706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.218841 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:36.218953 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:36.219072 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 04:56:36.219165 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 04:56:36.219304 13991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:56:36.219352 13991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:56:36.219688 13980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } } }
I20260928 04:56:36.219772 13980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:56:36.243929 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } attempt: 3
Sep 28 04:56:36 dist-test-slave-m39l krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:56:36.253015 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.244321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55314 (local address 127.0.98.60:42983)
0928 04:56:36.244371 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:36.244373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:36.244382 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:56:36.244411 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:36.244412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:36.244421 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:36.244494 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:36.244860 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.245619 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.246658 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.247017 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.247088 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.250000 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:36.250001 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:36.250001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:36.250009 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:36.251772 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.252183 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.252184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.252184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.252247 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:36.252458 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:36.252459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:36.252459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:36.252602 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:36.252683 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:36.252837 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 04:56:36.252921 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:56:36.253026 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:36.244220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42983 (local address 127.0.0.1:55314)
0928 04:56:36.244271 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:36.244283 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:36.244322 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:36.244511 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:36.244511 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:36.244721 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:36.244778 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:36.244780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.245719 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.245720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:36.246570 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:36.246573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:36.247097 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:36.247098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:36.247161 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:36.247731 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:36.247736 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:36.249910 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:36.251864 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:36.251865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:36.251866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:36.252103 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:36.252320 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:36.252321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:36.252321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:36.252391 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:36.252674 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:36.252675 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:36.252728 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 04:56:36.252801 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 04:56:36.252909 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 04:56:36.255247 13878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4: Bootstrap starting.
I20260928 04:56:36.256029 13878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:36.256328 13878 log.cc:824] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:36.257230 13878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4: No bootstrap required, opened a new log
I20260928 04:56:36.259829 13878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } }
I20260928 04:56:36.260021 13878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:36.260089 13878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14290cc0eaa5495b86d709bda88c3fa4, State: Initialized, Role: FOLLOWER
I20260928 04:56:36.260277 13878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } }
I20260928 04:56:36.260629 13994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } } }
I20260928 04:56:36.260731 13994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:56:36.260692 13878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 04:56:36.262838 14005 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:56:36.262908 14005 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:56:36.265177 13810 master_runner.cc:263] No leader master found from master 14290cc0eaa5495b86d709bda88c3fa4
I20260928 04:56:36.265237 13810 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 04:56:36.340838 13756 tablet.cc:2369] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:56:36.407845 13906 tablet.cc:2369] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:56:36.497898 13831 tablet.cc:2369] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:56:36.520515 13980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:56:36.520678 13980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } }
I20260928 04:56:36.521028 13980 leader_election.cc:288] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14290cc0eaa5495b86d709bda88c3fa4 (127.0.98.61:38193), 965cd4de176d483da7fcc0a5302526d6 (127.0.98.60:42983)
I20260928 04:56:36.521409 13852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63122aabe8d1411a895ee5576aab4b15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14290cc0eaa5495b86d709bda88c3fa4" is_pre_election: true
I20260928 04:56:36.521386 13927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63122aabe8d1411a895ee5576aab4b15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "965cd4de176d483da7fcc0a5302526d6" is_pre_election: true
I20260928 04:56:36.521612 13852 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63122aabe8d1411a895ee5576aab4b15 in term 0.
I20260928 04:56:36.521628 13927 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63122aabe8d1411a895ee5576aab4b15 in term 0.
I20260928 04:56:36.521807 13752 leader_election.cc:302] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14290cc0eaa5495b86d709bda88c3fa4, 63122aabe8d1411a895ee5576aab4b15; no voters:
I20260928 04:56:36.521977 13980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:56:36.522082 13980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:56:36.522123 13980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:36.523036 13980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } }
I20260928 04:56:36.523214 13980 leader_election.cc:288] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [CANDIDATE]: Term 1 election: Requested vote from peers 14290cc0eaa5495b86d709bda88c3fa4 (127.0.98.61:38193), 965cd4de176d483da7fcc0a5302526d6 (127.0.98.60:42983)
I20260928 04:56:36.523420 13852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63122aabe8d1411a895ee5576aab4b15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14290cc0eaa5495b86d709bda88c3fa4"
I20260928 04:56:36.523418 13927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63122aabe8d1411a895ee5576aab4b15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "965cd4de176d483da7fcc0a5302526d6"
I20260928 04:56:36.523530 13927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:36.523530 13852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:36.524324 13927 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63122aabe8d1411a895ee5576aab4b15 in term 1.
I20260928 04:56:36.524413 13852 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63122aabe8d1411a895ee5576aab4b15 in term 1.
I20260928 04:56:36.524540 13753 leader_election.cc:302] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63122aabe8d1411a895ee5576aab4b15, 965cd4de176d483da7fcc0a5302526d6; no voters:
I20260928 04:56:36.524660 13980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:56:36.524823 13980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [term 1 LEADER]: Becoming Leader. State: Replica: 63122aabe8d1411a895ee5576aab4b15, State: Running, Role: LEADER
I20260928 04:56:36.524946 13980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } }
I20260928 04:56:36.525394 14007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63122aabe8d1411a895ee5576aab4b15. Latest consensus state: current_term: 1 leader_uuid: "63122aabe8d1411a895ee5576aab4b15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } } }
I20260928 04:56:36.525502 14007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [sys.catalog]: This master's current role is: LEADER
I20260928 04:56:36.525735 14009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:56:36.525936 14009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:56:36.527649 13852 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4 [term 1 FOLLOWER]: Refusing update from remote peer 63122aabe8d1411a895ee5576aab4b15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:56:36.527769 13927 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6 [term 1 FOLLOWER]: Refusing update from remote peer 63122aabe8d1411a895ee5576aab4b15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:56:36.528031 14007 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:56:36.528200 13980 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:56:36.529336 13994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63122aabe8d1411a895ee5576aab4b15. Latest consensus state: current_term: 1 leader_uuid: "63122aabe8d1411a895ee5576aab4b15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } } }
I20260928 04:56:36.529450 13994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:56:36.529441 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63122aabe8d1411a895ee5576aab4b15. Latest consensus state: current_term: 1 leader_uuid: "63122aabe8d1411a895ee5576aab4b15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } } }
I20260928 04:56:36.529536 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:56:36.530622 14007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63122aabe8d1411a895ee5576aab4b15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } } }
I20260928 04:56:36.530680 14012 mvcc.cc:205] Tried to move back new op lower bound from 7334180440174305280 to 7334180440166989824. Current Snapshot: MvccSnapshot[applied={T|T < 7334180440174305280}]
I20260928 04:56:36.530649 13980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63122aabe8d1411a895ee5576aab4b15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } } }
I20260928 04:56:36.530750 14010 mvcc.cc:205] Tried to move back new op lower bound from 7334180440174305280 to 7334180440166989824. Current Snapshot: MvccSnapshot[applied={T|T < 7334180440174305280 or (T in {7334180440174305280})}]
I20260928 04:56:36.530763 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63122aabe8d1411a895ee5576aab4b15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } } }
I20260928 04:56:36.530730 14007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [sys.catalog]: This master's current role is: LEADER
I20260928 04:56:36.530833 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:56:36.530787 14009 catalog_manager.cc:1347] Generated new cluster ID: 880a128fbfcb4616a0f252c62f4d0427
I20260928 04:56:36.530866 14009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:56:36.531213 14011 mvcc.cc:205] Tried to move back new op lower bound from 7334180440174305280 to 7334180440166989824. Current Snapshot: MvccSnapshot[applied={T|T < 7334180440174305280}]
I20260928 04:56:36.531173 13994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63122aabe8d1411a895ee5576aab4b15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63122aabe8d1411a895ee5576aab4b15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33783 } } peers { permanent_uuid: "14290cc0eaa5495b86d709bda88c3fa4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38193 } } peers { permanent_uuid: "965cd4de176d483da7fcc0a5302526d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42983 } } }
I20260928 04:56:36.531276 13994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:56:36.530797 13980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15 [sys.catalog]: This master's current role is: LEADER
I20260928 04:56:36.547827 14009 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:56:36.548461 14009 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:56:36.555225 14009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63122aabe8d1411a895ee5576aab4b15: Generated new TSK 0
I20260928 04:56:36.555539 14009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:56:37.203125 13970 catalog_manager.cc:1259] Loaded cluster ID: 880a128fbfcb4616a0f252c62f4d0427
I20260928 04:56:37.203207 13970 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6: loading cluster ID for follower catalog manager: success
I20260928 04:56:37.204123 13970 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6: acquiring CA information for follower catalog manager: success
I20260928 04:56:37.204457 13970 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 965cd4de176d483da7fcc0a5302526d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:56:37.263367 14005 catalog_manager.cc:1259] Loaded cluster ID: 880a128fbfcb4616a0f252c62f4d0427
I20260928 04:56:37.263451 14005 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4: loading cluster ID for follower catalog manager: success
I20260928 04:56:37.264369 14005 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4: acquiring CA information for follower catalog manager: success
I20260928 04:56:37.264736 14005 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 14290cc0eaa5495b86d709bda88c3fa4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:56:37.616727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 13735
I20260928 04:56:37.621855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 13810
I20260928 04:56:37.627276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 13885
2026-09-28T04:56:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3104 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:56:37 dist-test-slave-m39l krb5kdc[14025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:56:37 dist-test-slave-m39l krb5kdc[14025](info): set up 2 sockets
Sep 28 04:56:37 dist-test-slave-m39l krb5kdc[14025](info): commencing operation
krb5kdc: starting...
W20260928 04:56:38.714751 392 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:56:38 dist-test-slave-m39l krb5kdc[14025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:56:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:56:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:56:38.850477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40315
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:56:38.938776 14038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:38.939056 14038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:38.939096 14038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:38.940333 14038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:56:38.940419 14038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:38.940444 14038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:38.940476 14038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:56:38.940508 14038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:56:38.942077 14038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:38.942392 14038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:38.942678 14038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:38.948115 14043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:38.948129 14046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:38.948124 14044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:38.948316 14038 server_base.cc:1034] running on GCE node
I20260928 04:56:38.948683 14038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:38.949035 14038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:38.950235 14038 hybrid_clock.cc:648] HybridClock initialized: now 1790571398950212 us; error 46 us; skew 500 ppm
Sep 28 04:56:38 dist-test-slave-m39l krb5kdc[14025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:38.953298 14038 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:56:38.953974 14038 webserver.cc:460] Webserver started at http://127.0.98.62:37193/ using document root <none> and password file <none>
I20260928 04:56:38.954246 14038 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:38.954329 14038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:38.954559 14038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:56:38.955879 14038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "aef09778c75a4a3399d2899f7ea837b6"
format_stamp: "Formatted at 2026-09-28 04:56:38 on dist-test-slave-m39l"
server_key: "03c7e12e3020d3edd57ce2fabbbca705"
server_key_iv: "78d268c46f2c7d450bc218467855e459"
server_key_version: "encryptionkey@0"
I20260928 04:56:38.956307 14038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "aef09778c75a4a3399d2899f7ea837b6"
format_stamp: "Formatted at 2026-09-28 04:56:38 on dist-test-slave-m39l"
server_key: "03c7e12e3020d3edd57ce2fabbbca705"
server_key_iv: "78d268c46f2c7d450bc218467855e459"
server_key_version: "encryptionkey@0"
I20260928 04:56:38.957911 14038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:56:38.958927 14053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:38.959121 14038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:56:38.959242 14038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "aef09778c75a4a3399d2899f7ea837b6"
format_stamp: "Formatted at 2026-09-28 04:56:38 on dist-test-slave-m39l"
server_key: "03c7e12e3020d3edd57ce2fabbbca705"
server_key_iv: "78d268c46f2c7d450bc218467855e459"
server_key_version: "encryptionkey@0"
I20260928 04:56:38.959328 14038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:38.968938 14038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:38.971635 14038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:38.971835 14038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:38.976074 14038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40315
I20260928 04:56:38.976140 14105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40315 every 8 connection(s)
I20260928 04:56:38.976529 14038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:56:38.977504 14106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:38.980022 14106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6: Bootstrap starting.
I20260928 04:56:38.980645 14106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:38.980949 14106 log.cc:824] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:38.981684 14106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6: No bootstrap required, opened a new log
I20260928 04:56:38.983415 14106 raft_consensus.cc:348] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef09778c75a4a3399d2899f7ea837b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40315 } }
I20260928 04:56:38.983582 14106 raft_consensus.cc:374] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:38.983606 14106 raft_consensus.cc:729] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aef09778c75a4a3399d2899f7ea837b6, State: Initialized, Role: FOLLOWER
I20260928 04:56:38.983794 14106 consensus_queue.cc:260] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef09778c75a4a3399d2899f7ea837b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40315 } }
I20260928 04:56:38.983858 14106 raft_consensus.cc:388] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:56:38.983881 14106 raft_consensus.cc:482] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:56:38.983950 14106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:38.984524 14106 raft_consensus.cc:504] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef09778c75a4a3399d2899f7ea837b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40315 } }
I20260928 04:56:38.984620 14106 leader_election.cc:302] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aef09778c75a4a3399d2899f7ea837b6; no voters:
I20260928 04:56:38.984822 14106 leader_election.cc:288] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:56:38.984898 14109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:56:38.985093 14106 sys_catalog.cc:564] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:56:38.985118 14109 raft_consensus.cc:686] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6 [term 1 LEADER]: Becoming Leader. State: Replica: aef09778c75a4a3399d2899f7ea837b6, State: Running, Role: LEADER
I20260928 04:56:38.985268 14109 consensus_queue.cc:237] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef09778c75a4a3399d2899f7ea837b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40315 } }
I20260928 04:56:38.985863 14110 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aef09778c75a4a3399d2899f7ea837b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef09778c75a4a3399d2899f7ea837b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40315 } } }
I20260928 04:56:38.985980 14110 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6 [sys.catalog]: This master's current role is: LEADER
I20260928 04:56:38.986270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 14038
I20260928 04:56:38.986374 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:56:38.986644 392 external_mini_cluster.cc:1428] Setting key 29edcb041a0af9c7ff56c8d091968d2f
I20260928 04:56:38.986265 14111 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aef09778c75a4a3399d2899f7ea837b6. Latest consensus state: current_term: 1 leader_uuid: "aef09778c75a4a3399d2899f7ea837b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef09778c75a4a3399d2899f7ea837b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40315 } } }
I20260928 04:56:38.986919 14111 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6 [sys.catalog]: This master's current role is: LEADER
W20260928 04:56:38.987779 14124 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:56:38.987839 14124 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:56:38.987897 14123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:56:38.988060 14123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:56:38.989701 14123 catalog_manager.cc:1347] Generated new cluster ID: 38ad058b571b42069c7c1743221e98b3
I20260928 04:56:38.989763 14123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:56:38 dist-test-slave-m39l krb5kdc[14025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:38.997690 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:38.987110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42148 (local address 127.0.98.62:40315)
0928 04:56:38.987265 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:38.987274 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:38.987314 (+ 40us) server_negotiation.cc:407] Connection header received
0928 04:56:38.987464 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:38.987469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:38.987514 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:38.987675 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:38.988481 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:38.989735 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:38.990622 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:38.991054 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:38.991202 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:38.993541 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:38.993545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:38.993546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:38.993560 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:38.995955 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:38.996358 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:38.996361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:38.996361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:38.996490 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:38.996713 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:38.996714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:38.996714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:38.997001 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:38.997175 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:38.997310 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 04:56:38.997466 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260928 04:56:38.998718 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:56:39.008380 14123 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:56:39.009210 14123 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:56:39.025918 14123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aef09778c75a4a3399d2899f7ea837b6: Generated new TSK 0
I20260928 04:56:39.026156 14123 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:56:39.052932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 14038
2026-09-28T04:56:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1424 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:56:39 dist-test-slave-m39l krb5kdc[14133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:56:39 dist-test-slave-m39l krb5kdc[14133](info): set up 2 sockets
Sep 28 04:56:39 dist-test-slave-m39l krb5kdc[14133](info): commencing operation
krb5kdc: starting...
W20260928 04:56:40.158804 392 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:56:40 dist-test-slave-m39l krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:56:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:56:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:56:40.290553 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42115
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:56:40.375309 14146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:40.375578 14146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:40.375617 14146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:40.376863 14146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:56:40.376940 14146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:40.376966 14146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:40.376998 14146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:56:40.377022 14146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:56:40.378824 14146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:40.379138 14146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:40.379405 14146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:40.384572 14154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:40.384589 14152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:40.384642 14151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:40.385082 14146 server_base.cc:1034] running on GCE node
I20260928 04:56:40.385357 14146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:40.385684 14146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:40.386857 14146 hybrid_clock.cc:648] HybridClock initialized: now 1790571400386840 us; error 36 us; skew 500 ppm
Sep 28 04:56:40 dist-test-slave-m39l krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:40.389510 14146 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:56:40.390004 14146 webserver.cc:460] Webserver started at http://127.0.98.62:43279/ using document root <none> and password file <none>
I20260928 04:56:40.390262 14146 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:40.390311 14146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:40.390476 14146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:56:40.391629 14146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "db67c836b4064d32b79ddd9a2437ca09"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "39789982af71f61f38f2a6c1a8f22bfb"
server_key_iv: "7ea2e28ee845080243aa99e97ec07296"
server_key_version: "encryptionkey@0"
I20260928 04:56:40.391997 14146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "db67c836b4064d32b79ddd9a2437ca09"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "39789982af71f61f38f2a6c1a8f22bfb"
server_key_iv: "7ea2e28ee845080243aa99e97ec07296"
server_key_version: "encryptionkey@0"
I20260928 04:56:40.393419 14146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:56:40.394219 14161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:40.394428 14146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:56:40.394550 14146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "db67c836b4064d32b79ddd9a2437ca09"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "39789982af71f61f38f2a6c1a8f22bfb"
server_key_iv: "7ea2e28ee845080243aa99e97ec07296"
server_key_version: "encryptionkey@0"
I20260928 04:56:40.394639 14146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:40.406272 14146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:40.408918 14146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:40.409147 14146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:40.413573 14146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42115
I20260928 04:56:40.413618 14213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42115 every 8 connection(s)
I20260928 04:56:40.414047 14146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:56:40.415189 14214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:40.415598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 14146
I20260928 04:56:40.415712 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:56:40.415911 392 external_mini_cluster.cc:1428] Setting key 1352b3a8855bdc3512d88ceb82d801d1
I20260928 04:56:40.418586 14214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09: Bootstrap starting.
I20260928 04:56:40.419250 14214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:40.419565 14214 log.cc:824] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:40.420336 14214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09: No bootstrap required, opened a new log
I20260928 04:56:40.422253 14214 raft_consensus.cc:348] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db67c836b4064d32b79ddd9a2437ca09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260928 04:56:40.422439 14214 raft_consensus.cc:374] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:40.422524 14214 raft_consensus.cc:729] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db67c836b4064d32b79ddd9a2437ca09, State: Initialized, Role: FOLLOWER
I20260928 04:56:40.422724 14214 consensus_queue.cc:260] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db67c836b4064d32b79ddd9a2437ca09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260928 04:56:40.422832 14214 raft_consensus.cc:388] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:56:40.422878 14214 raft_consensus.cc:482] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:56:40.422930 14214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:40.423763 14214 raft_consensus.cc:504] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db67c836b4064d32b79ddd9a2437ca09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
Sep 28 04:56:40 dist-test-slave-m39l krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:40.423890 14214 leader_election.cc:302] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db67c836b4064d32b79ddd9a2437ca09; no voters:
I20260928 04:56:40.424069 14214 leader_election.cc:288] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:56:40.424103 14219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:56:40.424225 14219 raft_consensus.cc:686] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 1 LEADER]: Becoming Leader. State: Replica: db67c836b4064d32b79ddd9a2437ca09, State: Running, Role: LEADER
I20260928 04:56:40.424360 14214 sys_catalog.cc:564] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:56:40.424367 14219 consensus_queue.cc:237] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db67c836b4064d32b79ddd9a2437ca09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260928 04:56:40.424952 14220 sys_catalog.cc:455] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db67c836b4064d32b79ddd9a2437ca09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db67c836b4064d32b79ddd9a2437ca09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } } }
I20260928 04:56:40.424973 14221 sys_catalog.cc:455] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db67c836b4064d32b79ddd9a2437ca09. Latest consensus state: current_term: 1 leader_uuid: "db67c836b4064d32b79ddd9a2437ca09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db67c836b4064d32b79ddd9a2437ca09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } } }
I20260928 04:56:40.425122 14220 sys_catalog.cc:458] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [sys.catalog]: This master's current role is: LEADER
I20260928 04:56:40.425132 14221 sys_catalog.cc:458] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [sys.catalog]: This master's current role is: LEADER
I20260928 04:56:40.425781 14228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:56:40.426672 14228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:56:40.428547 14228 catalog_manager.cc:1347] Generated new cluster ID: 61c23d17247a4d248c8a3627177a923c
I20260928 04:56:40.428617 14228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:56:40.429524 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:40.416580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.0.98.62:42115)
0928 04:56:40.416798 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:40.416807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:40.416843 (+ 36us) server_negotiation.cc:407] Connection header received
0928 04:56:40.417326 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:40.417333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:40.417400 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:40.417555 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:40.418411 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:40.419745 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:40.420766 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:40.421258 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:40.421411 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:40.424497 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:40.424502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:40.424502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:40.424518 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:40.426889 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:40.427547 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:40.427550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:40.427550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:40.427663 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:40.427910 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:40.427910 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:40.427911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:40.428418 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:40.428604 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:40.429144 (+ 540us) server_negotiation.cc:299] Negotiation successful
0928 04:56:40.429298 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":107,"threads_started":1}
I20260928 04:56:40.439568 14228 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:56:40.440374 14228 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:56:40.444607 14228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09: Generated new TSK 0
I20260928 04:56:40.444847 14228 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:56:40.474473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:56:40.561087 14240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:40.561383 14240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:40.561430 14240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:40.562814 14240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:40.562887 14240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:40.562934 14240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:56:40.564550 14240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:40.564857 14240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:40.565125 14240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:40.570647 14245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:40.570685 14246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:40.570652 14248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:40.570811 14240 server_base.cc:1034] running on GCE node
I20260928 04:56:40.571218 14240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:40.571516 14240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:40.572702 14240 hybrid_clock.cc:648] HybridClock initialized: now 1790571400572699 us; error 38 us; skew 500 ppm
Sep 28 04:56:40 dist-test-slave-m39l krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:40.575577 14240 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:56:40.576064 14240 webserver.cc:460] Webserver started at http://127.0.98.1:33569/ using document root <none> and password file <none>
I20260928 04:56:40.576331 14240 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:40.576383 14240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:40.576548 14240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:56:40.577715 14240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "945a0d946b324194876eeb161ac802f7"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "7df215aeea36046694f484d3e169fd69"
server_key_iv: "3f57299522ccb91183d627b1cb0d1321"
server_key_version: "encryptionkey@0"
I20260928 04:56:40.578097 14240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "945a0d946b324194876eeb161ac802f7"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "7df215aeea36046694f484d3e169fd69"
server_key_iv: "3f57299522ccb91183d627b1cb0d1321"
server_key_version: "encryptionkey@0"
I20260928 04:56:40.579670 14240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:56:40.580525 14255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:40.580767 14240 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:56:40.580873 14240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "945a0d946b324194876eeb161ac802f7"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "7df215aeea36046694f484d3e169fd69"
server_key_iv: "3f57299522ccb91183d627b1cb0d1321"
server_key_version: "encryptionkey@0"
I20260928 04:56:40.580958 14240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:40.594269 14240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:40.596659 14240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:40.596784 14240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:40.596969 14240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:56:40.597314 14240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:56:40.597395 14240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:40.597443 14240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:56:40.597471 14240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:40.603402 14240 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41023
I20260928 04:56:40.603490 14368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41023 every 8 connection(s)
I20260928 04:56:40.603843 14240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:56:40.610129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 14240
I20260928 04:56:40.610255 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:56:40.610448 392 external_mini_cluster.cc:1428] Setting key 57d83f84c01c2e4cbedeaef9cb43d743
Sep 28 04:56:40 dist-test-slave-m39l krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:40.614768 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:40.604704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51859 (local address 127.0.98.62:42115)
0928 04:56:40.604779 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:40.604781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:40.604957 (+ 176us) server_negotiation.cc:407] Connection header received
0928 04:56:40.605514 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:40.605516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:40.605527 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:40.605602 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:40.606714 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:40.607229 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:40.608246 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:40.608460 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:40.608521 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:40.611400 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:40.611401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:40.611402 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:40.611411 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:40.613082 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:40.613685 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:40.613686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:40.613686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:40.613742 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:40.614048 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:40.614049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:40.614049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:40.614163 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:40.614215 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:40.614592 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 04:56:40.614698 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 04:56:40.615145 14371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:40.604652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:51859)
0928 04:56:40.604819 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:40.604864 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:40.605364 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:40.605685 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:40.605689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:40.606316 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:40.606605 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:40.606612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:40.607313 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:40.607314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:40.608160 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:40.608162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:40.608538 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:40.608539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:40.608643 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:40.609237 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:40.609246 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:40.611303 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:40.613169 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:40.613178 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:40.613181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:40.613544 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:40.613817 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:40.613818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:40.613819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:40.613980 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:40.614232 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:40.614239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:40.614426 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 04:56:40.614572 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 04:56:40.614777 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260928 04:56:40.615808 14369 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42115
I20260928 04:56:40.615937 14369 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:40.616132 14369 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260928 04:56:40.616690 14178 ts_manager.cc:194] Registered new tserver with Master: 945a0d946b324194876eeb161ac802f7 (127.0.98.1:41023)
I20260928 04:56:40.617348 14178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51859
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:56:40.642091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:56:40.727186 14374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:40.727401 14374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:40.727425 14374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:40.728618 14374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:40.728688 14374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:40.728717 14374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:56:40.730140 14374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:40.730350 14374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:40.730566 14374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:56:40.735940 14374 server_base.cc:1034] running on GCE node
W20260928 04:56:40.735940 14379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:40.735971 14382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:40.735952 14380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:40.736433 14374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:40.736790 14374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:40.737985 14374 hybrid_clock.cc:648] HybridClock initialized: now 1790571400737969 us; error 31 us; skew 500 ppm
Sep 28 04:56:40 dist-test-slave-m39l krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:40.740757 14374 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:56:40.741282 14374 webserver.cc:460] Webserver started at http://127.0.98.2:38791/ using document root <none> and password file <none>
I20260928 04:56:40.741549 14374 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:40.741624 14374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:40.741775 14374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:56:40.742944 14374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "d6476329b23445a9a17e750d5a99c15b"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "b206a72f3e4bb59fa2f4b1758232d597"
server_key_iv: "84e78ce9ff5f9e994773de53fae460c9"
server_key_version: "encryptionkey@0"
I20260928 04:56:40.743314 14374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d6476329b23445a9a17e750d5a99c15b"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "b206a72f3e4bb59fa2f4b1758232d597"
server_key_iv: "84e78ce9ff5f9e994773de53fae460c9"
server_key_version: "encryptionkey@0"
I20260928 04:56:40.744683 14374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:56:40.745393 14389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:40.745576 14374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:56:40.745667 14374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "d6476329b23445a9a17e750d5a99c15b"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "b206a72f3e4bb59fa2f4b1758232d597"
server_key_iv: "84e78ce9ff5f9e994773de53fae460c9"
server_key_version: "encryptionkey@0"
I20260928 04:56:40.745752 14374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:40.767639 14374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:40.770172 14374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:40.770356 14374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:40.770673 14374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:56:40.771006 14374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:56:40.771062 14374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:40.771104 14374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:56:40.771131 14374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:40.777606 14374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45959
I20260928 04:56:40.777710 14502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45959 every 8 connection(s)
I20260928 04:56:40.778050 14374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:56:40 dist-test-slave-m39l krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:40.787472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 14374
I20260928 04:56:40.787588 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:56:40.787814 392 external_mini_cluster.cc:1428] Setting key 982c8d0514619fb588de9b5fa818ffbd
I20260928 04:56:40.788283 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:40.778915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47187 (local address 127.0.98.62:42115)
0928 04:56:40.778964 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:40.778966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:40.779175 (+ 209us) server_negotiation.cc:407] Connection header received
0928 04:56:40.779703 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:40.779704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:40.779714 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:40.779766 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:40.780783 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:40.781288 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:40.782366 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:40.782647 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:40.782700 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:40.785090 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:40.785092 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:40.785092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:40.785099 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:40.786722 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:40.787108 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:40.787109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:40.787109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:40.787171 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:40.787441 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:40.787443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:40.787443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:40.787607 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:40.787680 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:40.788078 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 04:56:40.788180 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:56:40.788503 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:40.778887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:47187)
0928 04:56:40.779041 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:40.779080 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:40.779589 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:40.779833 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:40.779836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:40.780468 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:40.780694 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:40.780700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:40.781375 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:40.781376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:40.782279 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:40.782281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:40.782721 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:40.782722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:40.782835 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:40.783356 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:40.783366 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:40.785003 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:40.786794 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:40.786800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:40.786802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:40.787039 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:40.787234 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:40.787234 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:40.787235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:40.787356 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:40.787712 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:40.787720 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:40.787907 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 04:56:40.788032 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 04:56:40.788234 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260928 04:56:40.788955 14503 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42115
I20260928 04:56:40.789079 14503 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:40.789283 14503 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260928 04:56:40.789733 14178 ts_manager.cc:194] Registered new tserver with Master: d6476329b23445a9a17e750d5a99c15b (127.0.98.2:45959)
I20260928 04:56:40.790153 14178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47187
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:56:40.817251 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:56:40.900696 14508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:40.900918 14508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:40.900945 14508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:40.902230 14508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:40.902276 14508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:40.902307 14508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:56:40.903824 14508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:40.904042 14508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:40.904246 14508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:40.909576 14514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:40.909577 14516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:40.909571 14513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:40.910022 14508 server_base.cc:1034] running on GCE node
I20260928 04:56:40.910332 14508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:40.910642 14508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:40.911795 14508 hybrid_clock.cc:648] HybridClock initialized: now 1790571400911789 us; error 36 us; skew 500 ppm
Sep 28 04:56:40 dist-test-slave-m39l krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:40.918785 14508 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:56:40.919344 14508 webserver.cc:460] Webserver started at http://127.0.98.3:42907/ using document root <none> and password file <none>
I20260928 04:56:40.919606 14508 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:40.919684 14508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:40.919842 14508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:56:40.920964 14508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "1ca74b63eda84f3998d00a8327ca9906"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "cb02d9ea5fc0f70f99fe631afe0500ed"
server_key_iv: "af47ce2a2d714e9f8e7db995f289da0f"
server_key_version: "encryptionkey@0"
I20260928 04:56:40.921346 14508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1ca74b63eda84f3998d00a8327ca9906"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "cb02d9ea5fc0f70f99fe631afe0500ed"
server_key_iv: "af47ce2a2d714e9f8e7db995f289da0f"
server_key_version: "encryptionkey@0"
I20260928 04:56:40.922830 14508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:56:40.923671 14523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:40.923870 14508 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:40.923983 14508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "1ca74b63eda84f3998d00a8327ca9906"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "cb02d9ea5fc0f70f99fe631afe0500ed"
server_key_iv: "af47ce2a2d714e9f8e7db995f289da0f"
server_key_version: "encryptionkey@0"
I20260928 04:56:40.924062 14508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:40.972496 14508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:40.975291 14508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:40.975462 14508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:40.975736 14508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:56:40.976071 14508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:56:40.976125 14508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:40.976161 14508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:56:40.976189 14508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:40.982343 14508 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37699
I20260928 04:56:40.982398 14636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37699 every 8 connection(s)
I20260928 04:56:40.982764 14508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:56:40.983031 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 14508
I20260928 04:56:40.983122 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:56:40.983326 392 external_mini_cluster.cc:1428] Setting key e128f3c075eadd25b3d44930d42f2ac7
Sep 28 04:56:40 dist-test-slave-m39l krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:40.992183 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:40.983705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50509 (local address 127.0.98.62:42115)
0928 04:56:40.983755 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:40.983757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:40.983924 (+ 167us) server_negotiation.cc:407] Connection header received
0928 04:56:40.984490 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:40.984491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:40.984501 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:40.984547 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:40.985513 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:40.986047 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:40.987124 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:40.987323 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:40.987379 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:40.989462 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:40.989464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:40.989464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:40.989470 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:40.990903 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:40.991327 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:40.991327 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:40.991328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:40.991382 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:40.991618 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:40.991619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:40.991619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:40.991736 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:40.991793 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:40.992048 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 04:56:40.992113 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:56:40.992342 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:40.983621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:50509)
0928 04:56:40.983804 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:40.983847 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:40.984400 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:40.984656 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:40.984660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:40.985236 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:40.985435 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:40.985440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:40.986136 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:40.986137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:40.987048 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:40.987050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:40.987399 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:40.987400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:40.987500 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:40.987994 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:40.988003 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:40.989376 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:40.990984 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:40.990991 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:40.990992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:40.991256 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:40.991449 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:40.991450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:40.991450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:40.991554 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:40.991796 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:40.991801 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:40.991919 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 04:56:40.992002 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 04:56:40.992123 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":113,"threads_started":1}
I20260928 04:56:40.992878 14637 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42115
I20260928 04:56:40.993002 14637 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:40.993206 14637 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260928 04:56:40.993636 14178 ts_manager.cc:194] Registered new tserver with Master: 1ca74b63eda84f3998d00a8327ca9906 (127.0.98.3:37699)
I20260928 04:56:40.994050 14178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50509
I20260928 04:56:40.995304 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:56:41.002969 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:40.996683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.0.98.62:42115)
0928 04:56:40.996757 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:40.996759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:40.996861 (+ 102us) server_negotiation.cc:407] Connection header received
0928 04:56:40.996899 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:40.996899 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:40.996909 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:40.996970 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:40.997490 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:40.997991 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:40.998826 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:40.999005 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:40.999061 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.000222 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:41.000223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:41.000223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:41.000230 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:41.001683 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:41.002100 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:41.002101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:41.002101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:41.002155 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:41.002412 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:41.002413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:41.002413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:41.002618 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:41.002727 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:41.002835 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 04:56:41.002899 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 04:56:41.004810 14178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:56:41.006588 14178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:56:41.013541 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.009490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37699 (local address 127.0.0.1:41114)
0928 04:56:41.009944 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:41.009956 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:41.010003 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:41.010252 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:41.010253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:41.010255 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:41.010306 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.010307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.011567 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.011568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.012478 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.012480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.013198 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.013199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.013289 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.013311 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 04:56:41.013401 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 04:56:41.013459 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":60,"threads_started":1}
I20260928 04:56:41.013553 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.009764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.0.98.3:37699)
0928 04:56:41.009974 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:41.009978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:41.009991 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:56:41.010073 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:41.010075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:41.010089 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:41.010173 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:41.010386 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.011470 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.012563 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.013087 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.013183 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.013249 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:41.013420 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 04:56:41.013460 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":97,"threads_started":1}
I20260928 04:56:41.013713 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.009337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41023 (local address 127.0.0.1:40192)
0928 04:56:41.009593 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:41.009604 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:41.009662 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:41.010064 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:41.010064 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:41.010067 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:41.010182 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.010184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.011196 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.011197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.012520 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.012523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.013508 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.013509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.013566 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.013569 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:56:41.013614 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 04:56:41.013644 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":71,"threads_started":1}
I20260928 04:56:41.013715 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.009120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45959 (local address 127.0.0.1:33052)
0928 04:56:41.009275 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:41.009346 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:41.009426 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:41.009870 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:41.009873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:41.009875 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:41.009966 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.009968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.011401 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.011402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.012521 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.012523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.013458 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.013459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.013526 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.013529 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:56:41.013572 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 04:56:41.013624 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":8,"thread_start_us":54,"threads_started":1}
I20260928 04:56:41.013988 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.009498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.0.98.1:41023)
0928 04:56:41.009833 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:41.009837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:41.009847 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:56:41.009886 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:41.009889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:41.009903 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:41.009980 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:41.010266 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.011074 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.012627 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.013356 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.013429 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.013481 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:41.013835 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 04:56:41.013887 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":105,"threads_started":1}
I20260928 04:56:41.014357 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.009325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33052 (local address 127.0.98.2:45959)
0928 04:56:41.009656 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:41.009662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:41.009673 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:56:41.009719 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:41.009721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:41.009739 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:41.009829 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:41.010081 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.011296 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.012630 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.013356 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.014125 (+ 769us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.014194 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:41.014237 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 04:56:41.014278 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
I20260928 04:56:41.014367 14571 tablet_service.cc:1467] Processing CreateTablet for tablet 12c1cb5aaf484af4ae0a2919cce2666d (DEFAULT_TABLE table=test-table [id=858f1766ac01426ba98e4d9294c0dc1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:56:41.014812 14571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c1cb5aaf484af4ae0a2919cce2666d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:41.015002 14437 tablet_service.cc:1467] Processing CreateTablet for tablet 12c1cb5aaf484af4ae0a2919cce2666d (DEFAULT_TABLE table=test-table [id=858f1766ac01426ba98e4d9294c0dc1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:56:41.015297 14437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c1cb5aaf484af4ae0a2919cce2666d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:41.017472 14654 tablet_bootstrap.cc:492] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906: Bootstrap starting.
I20260928 04:56:41.017941 14303 tablet_service.cc:1467] Processing CreateTablet for tablet 12c1cb5aaf484af4ae0a2919cce2666d (DEFAULT_TABLE table=test-table [id=858f1766ac01426ba98e4d9294c0dc1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:56:41.018008 14655 tablet_bootstrap.cc:492] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b: Bootstrap starting.
I20260928 04:56:41.018275 14303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c1cb5aaf484af4ae0a2919cce2666d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:41.018290 14654 tablet_bootstrap.cc:654] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:41.018707 14655 tablet_bootstrap.cc:654] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:41.018911 14654 log.cc:824] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:41.019245 14655 log.cc:824] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:41.019739 14654 tablet_bootstrap.cc:492] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906: No bootstrap required, opened a new log
I20260928 04:56:41.019819 14654 ts_tablet_manager.cc:1397] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:56:41.020018 14655 tablet_bootstrap.cc:492] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b: No bootstrap required, opened a new log
I20260928 04:56:41.020081 14655 ts_tablet_manager.cc:1397] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:56:41.021483 14658 tablet_bootstrap.cc:492] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7: Bootstrap starting.
I20260928 04:56:41.022015 14658 tablet_bootstrap.cc:654] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:41.022024 14655 raft_consensus.cc:348] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } } peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } }
I20260928 04:56:41.022024 14654 raft_consensus.cc:348] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } } peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } }
I20260928 04:56:41.022233 14655 raft_consensus.cc:374] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:41.022238 14654 raft_consensus.cc:374] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:41.022291 14654 raft_consensus.cc:729] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ca74b63eda84f3998d00a8327ca9906, State: Initialized, Role: FOLLOWER
I20260928 04:56:41.022291 14655 raft_consensus.cc:729] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6476329b23445a9a17e750d5a99c15b, State: Initialized, Role: FOLLOWER
I20260928 04:56:41.022434 14655 consensus_queue.cc:260] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } } peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } }
I20260928 04:56:41.022439 14654 consensus_queue.cc:260] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } } peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } }
I20260928 04:56:41.022694 14658 log.cc:824] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:41.022747 14655 ts_tablet_manager.cc:1428] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:56:41.022747 14654 ts_tablet_manager.cc:1428] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:56:41.022871 14637 heartbeater.cc:503] Master 127.0.98.62:42115 was elected leader, sending a full tablet report...
I20260928 04:56:41.022931 14503 heartbeater.cc:503] Master 127.0.98.62:42115 was elected leader, sending a full tablet report...
I20260928 04:56:41.023516 14658 tablet_bootstrap.cc:492] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7: No bootstrap required, opened a new log
I20260928 04:56:41.023629 14658 ts_tablet_manager.cc:1397] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:56:41.025403 14658 raft_consensus.cc:348] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } } peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } }
I20260928 04:56:41.025592 14658 raft_consensus.cc:374] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:41.025627 14658 raft_consensus.cc:729] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 945a0d946b324194876eeb161ac802f7, State: Initialized, Role: FOLLOWER
I20260928 04:56:41.025743 14658 consensus_queue.cc:260] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } } peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } }
I20260928 04:56:41.026037 14658 ts_tablet_manager.cc:1428] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:56:41.026079 14369 heartbeater.cc:503] Master 127.0.98.62:42115 was elected leader, sending a full tablet report...
W20260928 04:56:41.028638 14504 tablet.cc:2369] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:56:41.104652 14370 tablet.cc:2369] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:56:41.170183 14659 raft_consensus.cc:482] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:56:41.170321 14659 raft_consensus.cc:504] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } } peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } }
I20260928 04:56:41.170866 14659 leader_election.cc:288] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 945a0d946b324194876eeb161ac802f7 (127.0.98.1:41023), 1ca74b63eda84f3998d00a8327ca9906 (127.0.98.3:37699)
I20260928 04:56:41.173732 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.171200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56471 (local address 127.0.98.1:41023)
0928 04:56:41.171255 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:41.171257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:41.171268 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:56:41.171301 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:41.171302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:41.171314 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:41.171373 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:41.171542 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.172049 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.173016 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.173373 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.173459 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.173470 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:41.173619 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 04:56:41.173666 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:56:41.173732 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.171064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41023 (local address 127.0.98.2:56471)
0928 04:56:41.171130 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:41.171144 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:41.171223 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:41.171393 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:41.171394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:41.171396 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:41.171466 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.171468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.172148 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.172149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.172920 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.172922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.173456 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.173457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.173543 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.173546 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:56:41.173614 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 04:56:41.173658 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 04:56:41.173919 14663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.171216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37699 (local address 127.0.98.2:46187)
0928 04:56:41.171393 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:41.171402 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:41.171451 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:41.171576 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:41.171576 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:41.171578 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:41.171654 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.171656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.172287 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.172288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.173089 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.173091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.173649 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.173650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.173722 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.173731 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 04:56:41.173794 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 04:56:41.173846 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":121,"threads_started":1}
I20260928 04:56:41.173908 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.171322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46187 (local address 127.0.98.3:37699)
0928 04:56:41.171369 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:41.171371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:41.171443 (+ 72us) server_negotiation.cc:407] Connection header received
0928 04:56:41.171511 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:41.171512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:41.171523 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:41.171574 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:41.171735 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.172197 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.173179 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.173560 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.173645 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.173654 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:41.173799 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 04:56:41.173839 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:56:41.174099 14323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c1cb5aaf484af4ae0a2919cce2666d" candidate_uuid: "d6476329b23445a9a17e750d5a99c15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "945a0d946b324194876eeb161ac802f7" is_pre_election: true
I20260928 04:56:41.174297 14323 raft_consensus.cc:2454] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6476329b23445a9a17e750d5a99c15b in term 0.
I20260928 04:56:41.174580 14392 leader_election.cc:302] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 945a0d946b324194876eeb161ac802f7, d6476329b23445a9a17e750d5a99c15b; no voters:
I20260928 04:56:41.174755 14659 raft_consensus.cc:2781] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:56:41.174744 14591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c1cb5aaf484af4ae0a2919cce2666d" candidate_uuid: "d6476329b23445a9a17e750d5a99c15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ca74b63eda84f3998d00a8327ca9906" is_pre_election: true
I20260928 04:56:41.174811 14659 raft_consensus.cc:482] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:56:41.174835 14659 raft_consensus.cc:3037] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:41.174891 14591 raft_consensus.cc:2454] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6476329b23445a9a17e750d5a99c15b in term 0.
I20260928 04:56:41.175604 14659 raft_consensus.cc:504] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } } peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } }
I20260928 04:56:41.175745 14659 leader_election.cc:288] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [CANDIDATE]: Term 1 election: Requested vote from peers 945a0d946b324194876eeb161ac802f7 (127.0.98.1:41023), 1ca74b63eda84f3998d00a8327ca9906 (127.0.98.3:37699)
I20260928 04:56:41.175899 14323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c1cb5aaf484af4ae0a2919cce2666d" candidate_uuid: "d6476329b23445a9a17e750d5a99c15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "945a0d946b324194876eeb161ac802f7"
I20260928 04:56:41.175940 14591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c1cb5aaf484af4ae0a2919cce2666d" candidate_uuid: "d6476329b23445a9a17e750d5a99c15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ca74b63eda84f3998d00a8327ca9906"
I20260928 04:56:41.176007 14323 raft_consensus.cc:3037] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:41.176010 14591 raft_consensus.cc:3037] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:41.176775 14591 raft_consensus.cc:2454] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6476329b23445a9a17e750d5a99c15b in term 1.
I20260928 04:56:41.176766 14323 raft_consensus.cc:2454] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6476329b23445a9a17e750d5a99c15b in term 1.
I20260928 04:56:41.176964 14392 leader_election.cc:302] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca74b63eda84f3998d00a8327ca9906, d6476329b23445a9a17e750d5a99c15b; no voters:
I20260928 04:56:41.177053 14659 raft_consensus.cc:2781] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:56:41.177217 14659 raft_consensus.cc:686] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [term 1 LEADER]: Becoming Leader. State: Replica: d6476329b23445a9a17e750d5a99c15b, State: Running, Role: LEADER
I20260928 04:56:41.177323 14659 consensus_queue.cc:237] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } } peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } }
I20260928 04:56:41.178221 14177 catalog_manager.cc:5495] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b reported cstate change: term changed from 0 to 1, leader changed from <none> to d6476329b23445a9a17e750d5a99c15b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d6476329b23445a9a17e750d5a99c15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } health_report { overall_health: UNKNOWN } } }
I20260928 04:56:41.195647 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.193110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33068 (local address 127.0.98.2:45959)
0928 04:56:41.193184 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:41.193186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:41.193205 (+ 19us) server_negotiation.cc:407] Connection header received
0928 04:56:41.193249 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:41.193250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:41.193270 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:41.193347 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:56:41.193493 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.193987 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.194833 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.195026 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.195105 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.195304 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:56:41.195406 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:56:41.195495 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:41.195525 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 04:56:41.195546 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 04:56:41.197698 14323 raft_consensus.cc:1261] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Refusing update from remote peer d6476329b23445a9a17e750d5a99c15b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:56:41.197759 14591 raft_consensus.cc:1261] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906 [term 1 FOLLOWER]: Refusing update from remote peer d6476329b23445a9a17e750d5a99c15b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:56:41.198108 14664 consensus_queue.cc:1035] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:56:41.198238 14659 consensus_queue.cc:1035] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:56:41.201023 14668 mvcc.cc:205] Tried to move back new op lower bound from 7334180459302502400 to 7334180459222880256. Current Snapshot: MvccSnapshot[applied={T|T < 7334180459302502400 or (T in {7334180459302502400})}]
I20260928 04:56:41.201723 14669 mvcc.cc:205] Tried to move back new op lower bound from 7334180459302502400 to 7334180459222880256. Current Snapshot: MvccSnapshot[applied={T|T < 7334180459302502400}]
I20260928 04:56:41.203014 14177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:56:41.203135 14177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:56:41.203933 14177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09: Sending DeleteTablet for 3 replicas of tablet 12c1cb5aaf484af4ae0a2919cce2666d
I20260928 04:56:41.204317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 14240
I20260928 04:56:41.204412 14303 tablet_service.cc:1514] Processing DeleteTablet for tablet 12c1cb5aaf484af4ae0a2919cce2666d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:56:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40192
I20260928 04:56:41.204845 14437 tablet_service.cc:1514] Processing DeleteTablet for tablet 12c1cb5aaf484af4ae0a2919cce2666d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:56:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33052
I20260928 04:56:41.205112 14571 tablet_service.cc:1514] Processing DeleteTablet for tablet 12c1cb5aaf484af4ae0a2919cce2666d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:56:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41114
I20260928 04:56:41.205129 14678 tablet_replica.cc:330] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b: stopping tablet replica
I20260928 04:56:41.205250 14678 raft_consensus.cc:2229] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:56:41.205371 14678 raft_consensus.cc:2258] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:56:41.205454 14679 tablet_replica.cc:330] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906: stopping tablet replica
I20260928 04:56:41.205608 14679 raft_consensus.cc:2229] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:56:41.205761 14679 raft_consensus.cc:2258] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:56:41.205857 14678 ts_tablet_manager.cc:1905] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:56:41.206252 14679 ts_tablet_manager.cc:1905] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:56:41.207829 14678 ts_tablet_manager.cc:1918] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:56:41.207911 14678 log.cc:1196] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/12c1cb5aaf484af4ae0a2919cce2666d
I20260928 04:56:41.208199 14678 ts_tablet_manager.cc:1939] T 12c1cb5aaf484af4ae0a2919cce2666d P d6476329b23445a9a17e750d5a99c15b: Deleting consensus metadata
I20260928 04:56:41.208578 14679 ts_tablet_manager.cc:1918] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:56:41.208648 14164 catalog_manager.cc:4841] TS d6476329b23445a9a17e750d5a99c15b (127.0.98.2:45959): tablet 12c1cb5aaf484af4ae0a2919cce2666d (table test-table [id=858f1766ac01426ba98e4d9294c0dc1e]) successfully deleted
I20260928 04:56:41.208765 14679 log.cc:1196] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/12c1cb5aaf484af4ae0a2919cce2666d
I20260928 04:56:41.209095 14679 ts_tablet_manager.cc:1939] T 12c1cb5aaf484af4ae0a2919cce2666d P 1ca74b63eda84f3998d00a8327ca9906: Deleting consensus metadata
I20260928 04:56:41.209528 14164 catalog_manager.cc:4841] TS 1ca74b63eda84f3998d00a8327ca9906 (127.0.98.3:37699): tablet 12c1cb5aaf484af4ae0a2919cce2666d (table test-table [id=858f1766ac01426ba98e4d9294c0dc1e]) successfully deleted
W20260928 04:56:41.212001 14164 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41023 (error 108)
I20260928 04:56:41.212188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 14374
I20260928 04:56:41.213133 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.212985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41023 (local address 127.0.0.1:40204)
0928 04:56:41.213048 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:41.213096 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 04:56:41.213348 14164 catalog_manager.cc:4584] TS 945a0d946b324194876eeb161ac802f7 (127.0.98.1:41023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12c1cb5aaf484af4ae0a2919cce2666d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41023: connect: Connection refused (error 111)
I20260928 04:56:41.217734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 14508
I20260928 04:56:41.222303 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 14146
I20260928 04:56:41.227934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42115
--webserver_interface=127.0.98.62
--webserver_port=43279
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:56:41.312222 14681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:41.312502 14681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:41.312546 14681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:41.313846 14681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:56:41.313935 14681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:41.313961 14681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:41.313995 14681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:56:41.314023 14681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:56:41.315757 14681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:41.316072 14681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:41.316323 14681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:41.325922 14687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:41.325937 14688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:41.326130 14690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:41.326288 14681 server_base.cc:1034] running on GCE node
I20260928 04:56:41.326563 14681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:41.326870 14681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:41.328016 14681 hybrid_clock.cc:648] HybridClock initialized: now 1790571401327975 us; error 55 us; skew 500 ppm
Sep 28 04:56:41 dist-test-slave-m39l krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:41.330770 14681 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:56:41.331218 14681 webserver.cc:460] Webserver started at http://127.0.98.62:43279/ using document root <none> and password file <none>
I20260928 04:56:41.331489 14681 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:41.331573 14681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:41.333024 14681 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:56:41.333802 14697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:41.334014 14681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:56:41.334128 14681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "db67c836b4064d32b79ddd9a2437ca09"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "39789982af71f61f38f2a6c1a8f22bfb"
server_key_iv: "7ea2e28ee845080243aa99e97ec07296"
server_key_version: "encryptionkey@0"
I20260928 04:56:41.334616 14681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:41.350340 14681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:41.352640 14681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:41.352815 14681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:41.356820 14681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42115
I20260928 04:56:41.356859 14749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42115 every 8 connection(s)
I20260928 04:56:41.357295 14681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:56:41.359062 14750 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:56:41.359787 14750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09: Bootstrap starting.
I20260928 04:56:41.363373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 14681
I20260928 04:56:41.363746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41023
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42115
--webserver_port=33569
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:56:41.369083 14750 log.cc:824] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:41.371254 14750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:56:41.371584 14750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09: Bootstrap complete.
I20260928 04:56:41.374006 14750 raft_consensus.cc:348] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db67c836b4064d32b79ddd9a2437ca09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260928 04:56:41.374251 14750 raft_consensus.cc:729] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db67c836b4064d32b79ddd9a2437ca09, State: Initialized, Role: FOLLOWER
I20260928 04:56:41.374436 14750 consensus_queue.cc:260] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db67c836b4064d32b79ddd9a2437ca09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260928 04:56:41.374564 14750 raft_consensus.cc:388] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:56:41.374652 14750 raft_consensus.cc:482] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:56:41.374707 14750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 1 FOLLOWER]: Advancing to term 2
I20260928 04:56:41.375825 14750 raft_consensus.cc:504] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db67c836b4064d32b79ddd9a2437ca09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260928 04:56:41.375954 14750 leader_election.cc:302] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db67c836b4064d32b79ddd9a2437ca09; no voters:
I20260928 04:56:41.376140 14750 leader_election.cc:288] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 04:56:41.376211 14754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 04:56:41.376380 14754 raft_consensus.cc:686] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 2 LEADER]: Becoming Leader. State: Replica: db67c836b4064d32b79ddd9a2437ca09, State: Running, Role: LEADER
I20260928 04:56:41.376418 14750 sys_catalog.cc:564] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:56:41.376478 14754 consensus_queue.cc:237] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db67c836b4064d32b79ddd9a2437ca09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260928 04:56:41.376773 14754 sys_catalog.cc:455] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db67c836b4064d32b79ddd9a2437ca09. Latest consensus state: current_term: 2 leader_uuid: "db67c836b4064d32b79ddd9a2437ca09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db67c836b4064d32b79ddd9a2437ca09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } } }
I20260928 04:56:41.376847 14754 sys_catalog.cc:458] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [sys.catalog]: This master's current role is: LEADER
I20260928 04:56:41.376776 14755 sys_catalog.cc:455] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "db67c836b4064d32b79ddd9a2437ca09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db67c836b4064d32b79ddd9a2437ca09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } } }
I20260928 04:56:41.376914 14755 sys_catalog.cc:458] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [sys.catalog]: This master's current role is: LEADER
I20260928 04:56:41.377571 14760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:56:41.378471 14760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:56:41.378611 14760 catalog_manager.cc:1259] Loaded cluster ID: 61c23d17247a4d248c8a3627177a923c
I20260928 04:56:41.378645 14760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:56:41.378890 14760 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 04:56:41.383574 14760 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 04:56:41.386958 14760 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:56:41.387223 14760 catalog_manager.cc:5879] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09: Loaded TSK: 0
I20260928 04:56:41.387432 14760 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 04:56:41.451606 14752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:41.451834 14752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:41.451862 14752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:41.453230 14752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:41.453280 14752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:41.453311 14752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:56:41.455027 14752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33569
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:41.455302 14752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:41.455587 14752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:41.461761 14778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:41.461779 14780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:41.462447 14777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:41.470906 14752 server_base.cc:1034] running on GCE node
I20260928 04:56:41.471297 14752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:41.471668 14752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:41.472857 14752 hybrid_clock.cc:648] HybridClock initialized: now 1790571401472825 us; error 54 us; skew 500 ppm
Sep 28 04:56:41 dist-test-slave-m39l krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:41.476574 14752 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:56:41.477293 14752 webserver.cc:460] Webserver started at http://127.0.98.1:33569/ using document root <none> and password file <none>
I20260928 04:56:41.477669 14752 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:41.477747 14752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:41.479709 14752 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:41.480683 14787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:41.480898 14752 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:41.481002 14752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "945a0d946b324194876eeb161ac802f7"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "7df215aeea36046694f484d3e169fd69"
server_key_iv: "3f57299522ccb91183d627b1cb0d1321"
server_key_version: "encryptionkey@0"
I20260928 04:56:41.481480 14752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:41.511461 14752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:41.514542 14752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:41.514699 14752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:41.514962 14752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:56:41.515494 14794 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 04:56:41.517961 14752 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 04:56:41.518039 14752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260928 04:56:41.518077 14752 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 04:56:41.519065 14752 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 04:56:41.519115 14752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:41.519302 14794 tablet_bootstrap.cc:492] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7: Bootstrap starting.
I20260928 04:56:41.526491 14752 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41023
I20260928 04:56:41.526578 14901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41023 every 8 connection(s)
I20260928 04:56:41.526935 14752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:56:41.531896 14794 log.cc:824] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:41.533200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 14752
I20260928 04:56:41.533533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45959
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42115
--webserver_port=38791
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:56:41.536159 14794 tablet_bootstrap.cc:492] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:56:41.536495 14794 tablet_bootstrap.cc:492] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7: Bootstrap complete.
Sep 28 04:56:41 dist-test-slave-m39l krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:41.537829 14794 ts_tablet_manager.cc:1397] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.000s
I20260928 04:56:41.539563 14794 raft_consensus.cc:348] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } } peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } }
I20260928 04:56:41.539947 14794 raft_consensus.cc:729] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 945a0d946b324194876eeb161ac802f7, State: Initialized, Role: FOLLOWER
I20260928 04:56:41.540117 14794 consensus_queue.cc:260] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } } peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } }
I20260928 04:56:41.540477 14794 ts_tablet_manager.cc:1428] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:56:41.545338 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.527678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:56633)
0928 04:56:41.527875 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:41.527921 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:41.528376 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:41.531152 (+ 2776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:41.531157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:41.531546 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:41.531777 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.531784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.533077 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.533079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.534112 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.534115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.535135 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.535136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.535315 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.536000 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:41.536009 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:41.539329 (+ 3320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:41.542755 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:41.542768 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:41.542771 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:41.543181 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:41.543607 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:41.543609 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:41.543609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:41.543811 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:41.544446 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:41.544455 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:41.544658 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 04:56:41.544806 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 04:56:41.545049 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":113,"threads_started":1}
I20260928 04:56:41.545336 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.527785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56633 (local address 127.0.98.62:42115)
0928 04:56:41.530704 (+ 2919us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:41.530714 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:41.530753 (+ 39us) server_negotiation.cc:407] Connection header received
0928 04:56:41.530880 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:41.530886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:41.530933 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:41.531124 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:41.531914 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.532921 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.534211 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.534892 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.535010 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.539619 (+ 4609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:41.539625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:41.539625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:41.539641 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:41.542612 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:41.543316 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:41.543320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:41.543320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:41.543476 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:41.543914 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:41.543915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:41.543916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:41.544307 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:41.544401 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:41.544885 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 04:56:41.545090 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2854,"thread_start_us":105,"threads_started":1}
I20260928 04:56:41.546345 14902 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42115
I20260928 04:56:41.546478 14902 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:41.546749 14902 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260928 04:56:41.547765 14714 ts_manager.cc:194] Registered new tserver with Master: 945a0d946b324194876eeb161ac802f7 (127.0.98.1:41023)
I20260928 04:56:41.549108 14714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56633
I20260928 04:56:41.550309 14902 heartbeater.cc:503] Master 127.0.98.62:42115 was elected leader, sending a full tablet report...
Sep 28 04:56:41 dist-test-slave-m39l krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:56:41.560869 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.548780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40214 (local address 127.0.98.1:41023)
0928 04:56:41.548958 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:41.548963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:41.548973 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:56:41.549013 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:41.549015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:41.549029 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:41.549113 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:41.550991 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.552209 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.553562 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.554105 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.554178 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.557340 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:41.557354 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:41.557355 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:41.557366 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:41.559390 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:41.559947 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:41.559951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:41.559951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:41.560010 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:41.560263 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:41.560263 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:41.560264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:41.560424 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:41.560487 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:41.560706 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 04:56:41.560794 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260928 04:56:41.561075 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.548502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41023 (local address 127.0.0.1:40214)
0928 04:56:41.548685 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:41.548726 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:41.548799 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:41.549279 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:41.549286 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:41.550788 (+ 1502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:41.550875 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.550880 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.552361 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.552363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.553444 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.553447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.554255 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.554260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.554365 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.555061 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:41.555068 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:41.557211 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:41.559499 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:41.559510 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:41.559513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:41.559845 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:41.560106 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:41.560107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:41.560108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:41.560194 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:41.560494 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:41.560503 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:41.560573 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 04:56:41.560669 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 04:56:41.560827 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260928 04:56:41.561594 14834 tablet_service.cc:1514] Processing DeleteTablet for tablet 12c1cb5aaf484af4ae0a2919cce2666d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:56:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40214
I20260928 04:56:41.562176 14911 tablet_replica.cc:330] stopping tablet replica
I20260928 04:56:41.562296 14911 raft_consensus.cc:2229] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:56:41.562374 14911 raft_consensus.cc:2258] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:56:41.562736 14911 ts_tablet_manager.cc:1905] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:56:41.564713 14911 ts_tablet_manager.cc:1918] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:56:41.564787 14911 log.cc:1196] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/wal/wals/12c1cb5aaf484af4ae0a2919cce2666d
I20260928 04:56:41.565037 14911 ts_tablet_manager.cc:1939] T 12c1cb5aaf484af4ae0a2919cce2666d P 945a0d946b324194876eeb161ac802f7: Deleting consensus metadata
I20260928 04:56:41.565631 14700 catalog_manager.cc:4841] TS 945a0d946b324194876eeb161ac802f7 (127.0.98.1:41023): tablet 12c1cb5aaf484af4ae0a2919cce2666d (table test-table [id=858f1766ac01426ba98e4d9294c0dc1e]) successfully deleted
W20260928 04:56:41.634320 14907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:41.634646 14907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:41.634696 14907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:41.636045 14907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:41.636129 14907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:41.636184 14907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:56:41.638008 14907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38791
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:41.638327 14907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:41.638635 14907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:41.644369 14916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:41.644517 14907 server_base.cc:1034] running on GCE node
W20260928 04:56:41.644378 14917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:41.644383 14919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:41.644816 14907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:41.645097 14907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:41.646296 14907 hybrid_clock.cc:648] HybridClock initialized: now 1790571401646287 us; error 37 us; skew 500 ppm
Sep 28 04:56:41 dist-test-slave-m39l krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:41.649366 14907 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:56:41.649914 14907 webserver.cc:460] Webserver started at http://127.0.98.2:38791/ using document root <none> and password file <none>
I20260928 04:56:41.650199 14907 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:41.650303 14907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:41.652014 14907 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:41.652830 14926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:41.653074 14907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:41.653189 14907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "d6476329b23445a9a17e750d5a99c15b"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "b206a72f3e4bb59fa2f4b1758232d597"
server_key_iv: "84e78ce9ff5f9e994773de53fae460c9"
server_key_version: "encryptionkey@0"
I20260928 04:56:41.653595 14907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:41.686311 14907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:41.688891 14907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:41.689096 14907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:41.689427 14907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:56:41.689780 14907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:56:41.689836 14907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:41.689879 14907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:56:41.689906 14907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:41.696689 14907 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45959
I20260928 04:56:41.696725 15039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45959 every 8 connection(s)
I20260928 04:56:41.697129 14907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:56:41.700776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 14907
I20260928 04:56:41.701050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37699
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42115
--webserver_port=42907
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:56:41 dist-test-slave-m39l krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:41.708148 15042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.697941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:53733)
0928 04:56:41.698136 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:41.698183 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:41.698769 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:41.699001 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:41.699005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:41.699616 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:41.699824 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.699829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.700611 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.700612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.701723 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.701726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.702306 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.702308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.702485 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.703085 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:41.703096 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:41.704711 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:41.706712 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:41.706718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:41.706720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:41.706960 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:41.707193 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:41.707194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:41.707194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:41.707296 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:41.707614 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:41.707619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:41.707749 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 04:56:41.707835 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 04:56:41.707963 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260928 04:56:41.708151 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.697941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53733 (local address 127.0.98.62:42115)
0928 04:56:41.698035 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:41.698037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:41.698296 (+ 259us) server_negotiation.cc:407] Connection header received
0928 04:56:41.698861 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:41.698862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:41.698872 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:41.698942 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:41.699946 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.700453 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.701818 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.702177 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.702265 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.704829 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:41.704831 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:41.704831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:41.704841 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:41.706623 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:41.707038 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:41.707040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:41.707040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:41.707119 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:41.707364 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:41.707365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:41.707366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:41.707542 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:41.707622 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:41.707883 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 04:56:41.707988 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 04:56:41.708798 15040 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42115
I20260928 04:56:41.708900 15040 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:41.709098 15040 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260928 04:56:41.709506 14714 ts_manager.cc:194] Registered new tserver with Master: d6476329b23445a9a17e750d5a99c15b (127.0.98.2:45959)
I20260928 04:56:41.709847 14714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53733
W20260928 04:56:41.786598 15043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:41.786862 15043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:41.786903 15043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:41.788121 15043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:41.788203 15043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:41.788252 15043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:56:41.789784 15043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42907
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:41.790061 15043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:41.790309 15043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:41.795987 15049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:41.796002 15051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:41.796002 15048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:41.796337 15043 server_base.cc:1034] running on GCE node
I20260928 04:56:41.796550 15043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:41.796815 15043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:41.797987 15043 hybrid_clock.cc:648] HybridClock initialized: now 1790571401797966 us; error 39 us; skew 500 ppm
Sep 28 04:56:41 dist-test-slave-m39l krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:41.800619 15043 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:56:41.801081 15043 webserver.cc:460] Webserver started at http://127.0.98.3:42907/ using document root <none> and password file <none>
I20260928 04:56:41.801330 15043 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:41.801427 15043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:41.802912 15043 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:56:41.803506 15058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:41.803668 15043 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:56:41.803773 15043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "1ca74b63eda84f3998d00a8327ca9906"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "cb02d9ea5fc0f70f99fe631afe0500ed"
server_key_iv: "af47ce2a2d714e9f8e7db995f289da0f"
server_key_version: "encryptionkey@0"
I20260928 04:56:41.804149 15043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:41.817240 15043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:41.819288 15043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:41.819453 15043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:41.819695 15043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:56:41.820003 15043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:56:41.820056 15043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:41.820096 15043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:56:41.820132 15043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:41.826071 15043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37699
I20260928 04:56:41.826122 15171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37699 every 8 connection(s)
I20260928 04:56:41.826478 15043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:56:41.826725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 15043
Sep 28 04:56:41 dist-test-slave-m39l krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:41.834988 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.827253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.0.98.62:42115)
0928 04:56:41.827384 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:41.827386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:41.827529 (+ 143us) server_negotiation.cc:407] Connection header received
0928 04:56:41.827593 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:41.827594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:41.827606 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:41.827669 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:41.828492 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.829109 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.830110 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.830419 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.830527 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.832105 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:41.832106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:41.832106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:41.832114 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:41.833810 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:41.834150 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:41.834151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:41.834151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:41.834222 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:41.834414 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:41.834415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:41.834416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:41.834585 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:41.834728 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:41.834844 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 04:56:41.834913 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260928 04:56:41.837870 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.827249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36237 (local address 127.0.98.62:42115)
0928 04:56:41.827301 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:41.827303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:41.827563 (+ 260us) server_negotiation.cc:407] Connection header received
0928 04:56:41.828470 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:41.828534 (+ 64us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:41.828548 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:41.828621 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:41.829802 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.830433 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.832099 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.832400 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.832490 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.835115 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:41.835117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:41.835117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:41.835126 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:41.836549 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:41.836991 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:41.836992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:41.836992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:41.837041 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:41.837254 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:41.837255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:41.837255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:41.837371 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:41.837438 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:41.837732 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 04:56:41.837800 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:56:41.838090 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.827268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:36237)
0928 04:56:41.827440 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:41.827497 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:41.828266 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:41.828717 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:41.828721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:41.829418 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:41.829692 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.829700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.830672 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.830673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.831985 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.831988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.832520 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.832521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.832680 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.833199 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:41.833210 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:41.834999 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:41.836643 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:41.836650 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:41.836652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:41.836913 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:41.837097 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:41.837098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:41.837098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:41.837200 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:41.837466 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:41.837472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:41.837601 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 04:56:41.837698 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 04:56:41.837829 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260928 04:56:41.838661 15172 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42115
I20260928 04:56:41.838765 15172 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:41.838919 15172 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260928 04:56:41.839361 14714 ts_manager.cc:194] Registered new tserver with Master: 1ca74b63eda84f3998d00a8327ca9906 (127.0.98.3:37699)
I20260928 04:56:41.839792 14714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36237
I20260928 04:56:41.847103 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:56:41.850137 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.847587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.0.98.62:42115)
0928 04:56:41.847688 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:41.847690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:41.847700 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:56:41.847772 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:41.847773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:41.847810 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:41.847891 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:56:41.848047 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.848534 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.849336 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.849527 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.849612 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.849810 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:56:41.849914 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:56:41.850013 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:41.850030 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 04:56:41.850048 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 04:56:41.850534 14714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:56:41.852185 14714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:56:41.855046 14834 tablet_service.cc:1467] Processing CreateTablet for tablet cac57dce140c4f9e8b0bc89dfec15c91 (DEFAULT_TABLE table=test-table [id=b5acb62159e745c7ad680af5f908543f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:56:41.855291 14834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cac57dce140c4f9e8b0bc89dfec15c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:41.857168 14794 tablet_bootstrap.cc:492] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7: Bootstrap starting.
I20260928 04:56:41.857481 14794 tablet_bootstrap.cc:654] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:41.858397 14794 tablet_bootstrap.cc:492] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7: No bootstrap required, opened a new log
I20260928 04:56:41.858464 14794 ts_tablet_manager.cc:1397] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:41.858682 14794 raft_consensus.cc:348] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } }
I20260928 04:56:41.858788 14794 raft_consensus.cc:374] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:41.858847 14794 raft_consensus.cc:729] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 945a0d946b324194876eeb161ac802f7, State: Initialized, Role: FOLLOWER
I20260928 04:56:41.858908 14794 consensus_queue.cc:260] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } }
I20260928 04:56:41.859028 14794 ts_tablet_manager.cc:1428] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:41.859148 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.854458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37699 (local address 127.0.0.1:41126)
0928 04:56:41.854563 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:41.854578 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:41.854633 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:41.855018 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:41.855019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:41.855021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:41.855086 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.855087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.856237 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.856238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.857591 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.857595 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.858820 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.858821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.858913 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.858945 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 04:56:41.858985 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 04:56:41.859049 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 04:56:41.859549 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.854595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41126 (local address 127.0.98.3:37699)
0928 04:56:41.854785 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:41.854791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:41.854803 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:56:41.854851 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:41.854854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:41.854871 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:41.855197 (+ 326us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:41.855282 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.856117 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.857739 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.858700 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.859261 (+ 561us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.859353 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:41.859384 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 04:56:41.859429 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260928 04:56:41.859791 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.854586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45959 (local address 127.0.0.1:33084)
0928 04:56:41.854951 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:41.854962 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:41.855021 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:41.855306 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:41.855307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:41.855309 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:41.855363 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.855365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.856606 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.856607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.858047 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.858051 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.859515 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.859516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.859584 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.859589 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:56:41.859651 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 04:56:41.859709 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":90,"threads_started":1}
I20260928 04:56:41.860033 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.854738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33084 (local address 127.0.98.2:45959)
0928 04:56:41.854951 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:41.854956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:41.855027 (+ 71us) server_negotiation.cc:407] Connection header received
0928 04:56:41.855098 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:41.855100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:41.855115 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:41.855217 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:41.855445 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.856496 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.858177 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.859444 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.859516 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.859574 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:41.859905 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 04:56:41.859955 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":110,"threads_started":1}
I20260928 04:56:41.860376 15106 tablet_service.cc:1467] Processing CreateTablet for tablet cac57dce140c4f9e8b0bc89dfec15c91 (DEFAULT_TABLE table=test-table [id=b5acb62159e745c7ad680af5f908543f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:56:41.860757 15106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cac57dce140c4f9e8b0bc89dfec15c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:41.860726 14974 tablet_service.cc:1467] Processing CreateTablet for tablet cac57dce140c4f9e8b0bc89dfec15c91 (DEFAULT_TABLE table=test-table [id=b5acb62159e745c7ad680af5f908543f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:56:41.861037 14974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cac57dce140c4f9e8b0bc89dfec15c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:41.864557 15182 tablet_bootstrap.cc:492] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906: Bootstrap starting.
I20260928 04:56:41.865242 15182 tablet_bootstrap.cc:654] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:41.865768 15182 log.cc:824] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:41.866123 15183 tablet_bootstrap.cc:492] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b: Bootstrap starting.
I20260928 04:56:41.866680 15182 tablet_bootstrap.cc:492] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906: No bootstrap required, opened a new log
I20260928 04:56:41.866744 15182 ts_tablet_manager.cc:1397] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:56:41.866801 15183 tablet_bootstrap.cc:654] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:41.867280 15183 log.cc:824] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:41.868204 15183 tablet_bootstrap.cc:492] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b: No bootstrap required, opened a new log
I20260928 04:56:41.868266 15183 ts_tablet_manager.cc:1397] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:56:41.868584 15182 raft_consensus.cc:348] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } }
I20260928 04:56:41.868774 15182 raft_consensus.cc:374] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:41.868844 15182 raft_consensus.cc:729] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ca74b63eda84f3998d00a8327ca9906, State: Initialized, Role: FOLLOWER
I20260928 04:56:41.869000 15182 consensus_queue.cc:260] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } }
I20260928 04:56:41.869311 15182 ts_tablet_manager.cc:1428] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:56:41.869349 15172 heartbeater.cc:503] Master 127.0.98.62:42115 was elected leader, sending a full tablet report...
I20260928 04:56:41.870482 15183 raft_consensus.cc:348] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } }
I20260928 04:56:41.870699 15183 raft_consensus.cc:374] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:41.870754 15183 raft_consensus.cc:729] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6476329b23445a9a17e750d5a99c15b, State: Initialized, Role: FOLLOWER
I20260928 04:56:41.870884 15183 consensus_queue.cc:260] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } }
I20260928 04:56:41.871217 15040 heartbeater.cc:503] Master 127.0.98.62:42115 was elected leader, sending a full tablet report...
I20260928 04:56:41.871225 15183 ts_tablet_manager.cc:1428] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 04:56:41.876407 15186 raft_consensus.cc:482] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:56:41.876665 15186 raft_consensus.cc:504] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } }
I20260928 04:56:41.877097 15186 leader_election.cc:288] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6476329b23445a9a17e750d5a99c15b (127.0.98.2:45959), 945a0d946b324194876eeb161ac802f7 (127.0.98.1:41023)
I20260928 04:56:41.880452 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.877394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41023 (local address 127.0.98.3:52779)
0928 04:56:41.877606 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:41.877616 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:41.877667 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:41.877883 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:41.877884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:41.877885 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:41.877982 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.877984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.878733 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.878734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.879565 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.879567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.880182 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.880183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.880246 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.880250 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:56:41.880316 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 04:56:41.880357 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":28,"thread_start_us":146,"threads_started":1}
I20260928 04:56:41.880439 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.877593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52779 (local address 127.0.98.1:41023)
0928 04:56:41.877644 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:41.877647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:41.877658 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:56:41.877731 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:41.877732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:41.877756 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:41.877839 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:41.878043 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.878650 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.879664 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.880094 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.880182 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.880263 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:41.880323 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 04:56:41.880363 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:56:41.880555 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.877666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40203 (local address 127.0.98.2:45959)
0928 04:56:41.877720 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:41.877722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:41.877729 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:56:41.877780 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:41.877781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:41.877795 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:41.877866 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:41.878094 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.878694 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.879707 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.880145 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.880248 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.880259 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:41.880440 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 04:56:41.880488 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 04:56:41.880555 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.877253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45959 (local address 127.0.98.3:40203)
0928 04:56:41.877306 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:41.877320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:41.877426 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:41.877874 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:41.877874 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:41.877876 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:41.877997 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.877999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.878787 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.878788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.879606 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:41.879608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.880252 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.880253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:41.880348 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.880351 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:56:41.880431 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 04:56:41.880479 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 04:56:41.880853 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cac57dce140c4f9e8b0bc89dfec15c91" candidate_uuid: "1ca74b63eda84f3998d00a8327ca9906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "945a0d946b324194876eeb161ac802f7" is_pre_election: true
I20260928 04:56:41.881073 14856 raft_consensus.cc:2454] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ca74b63eda84f3998d00a8327ca9906 in term 0.
I20260928 04:56:41.881361 15061 leader_election.cc:302] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca74b63eda84f3998d00a8327ca9906, 945a0d946b324194876eeb161ac802f7; no voters:
I20260928 04:56:41.881484 15186 raft_consensus.cc:2781] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:56:41.881451 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cac57dce140c4f9e8b0bc89dfec15c91" candidate_uuid: "1ca74b63eda84f3998d00a8327ca9906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6476329b23445a9a17e750d5a99c15b" is_pre_election: true
I20260928 04:56:41.881541 15186 raft_consensus.cc:482] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:56:41.881572 15186 raft_consensus.cc:3037] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:41.881613 14994 raft_consensus.cc:2454] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ca74b63eda84f3998d00a8327ca9906 in term 0.
I20260928 04:56:41.882318 15186 raft_consensus.cc:504] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } }
I20260928 04:56:41.882458 15186 leader_election.cc:288] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [CANDIDATE]: Term 1 election: Requested vote from peers d6476329b23445a9a17e750d5a99c15b (127.0.98.2:45959), 945a0d946b324194876eeb161ac802f7 (127.0.98.1:41023)
I20260928 04:56:41.882599 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cac57dce140c4f9e8b0bc89dfec15c91" candidate_uuid: "1ca74b63eda84f3998d00a8327ca9906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6476329b23445a9a17e750d5a99c15b"
I20260928 04:56:41.882665 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cac57dce140c4f9e8b0bc89dfec15c91" candidate_uuid: "1ca74b63eda84f3998d00a8327ca9906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "945a0d946b324194876eeb161ac802f7"
I20260928 04:56:41.882695 14994 raft_consensus.cc:3037] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:41.882736 14856 raft_consensus.cc:3037] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:41.883586 14856 raft_consensus.cc:2454] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ca74b63eda84f3998d00a8327ca9906 in term 1.
I20260928 04:56:41.883586 14994 raft_consensus.cc:2454] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ca74b63eda84f3998d00a8327ca9906 in term 1.
I20260928 04:56:41.883744 15061 leader_election.cc:302] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca74b63eda84f3998d00a8327ca9906, 945a0d946b324194876eeb161ac802f7; no voters:
I20260928 04:56:41.883848 15186 raft_consensus.cc:2781] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:56:41.884039 15186 raft_consensus.cc:686] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [term 1 LEADER]: Becoming Leader. State: Replica: 1ca74b63eda84f3998d00a8327ca9906, State: Running, Role: LEADER
I20260928 04:56:41.884119 15186 consensus_queue.cc:237] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } }
I20260928 04:56:41.884927 14714 catalog_manager.cc:5495] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ca74b63eda84f3998d00a8327ca9906 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1ca74b63eda84f3998d00a8327ca9906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } health_report { overall_health: UNKNOWN } } }
I20260928 04:56:41.892092 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:41.889662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.0.98.3:37699)
0928 04:56:41.889703 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:41.889705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:41.889714 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:56:41.889775 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:41.889776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:41.889792 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:41.889858 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:56:41.890021 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.890623 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.891370 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:41.891585 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:41.891645 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:41.891792 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:56:41.891897 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:56:41.891962 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:41.892000 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 04:56:41.892022 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:56:41.894006 14994 raft_consensus.cc:1261] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b [term 1 FOLLOWER]: Refusing update from remote peer 1ca74b63eda84f3998d00a8327ca9906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:56:41.894016 14856 raft_consensus.cc:1261] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Refusing update from remote peer 1ca74b63eda84f3998d00a8327ca9906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:56:41.894356 15186 consensus_queue.cc:1035] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:56:41.894563 15189 consensus_queue.cc:1035] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:56:41.896781 15192 mvcc.cc:205] Tried to move back new op lower bound from 7334180462154952704 to 7334180462118064128. Current Snapshot: MvccSnapshot[applied={T|T < 7334180462154952704}]
I20260928 04:56:41.897033 15193 mvcc.cc:205] Tried to move back new op lower bound from 7334180462154952704 to 7334180462118064128. Current Snapshot: MvccSnapshot[applied={T|T < 7334180462154952704}]
I20260928 04:56:41.897522 15194 mvcc.cc:205] Tried to move back new op lower bound from 7334180462154952704 to 7334180462118064128. Current Snapshot: MvccSnapshot[applied={T|T < 7334180462154952704}]
I20260928 04:56:41.899879 14713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:56:41.899974 14713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:56:41.900883 14713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09: Sending DeleteTablet for 3 replicas of tablet cac57dce140c4f9e8b0bc89dfec15c91
I20260928 04:56:41.901286 14834 tablet_service.cc:1514] Processing DeleteTablet for tablet cac57dce140c4f9e8b0bc89dfec15c91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:56:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40214
I20260928 04:56:41.901306 14974 tablet_service.cc:1514] Processing DeleteTablet for tablet cac57dce140c4f9e8b0bc89dfec15c91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:56:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33084
I20260928 04:56:41.901345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 14752
I20260928 04:56:41.901391 14911 tablet_replica.cc:330] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7: stopping tablet replica
I20260928 04:56:41.901443 14911 raft_consensus.cc:2229] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:56:41.901495 14911 raft_consensus.cc:2258] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:56:41.903587 15201 tablet_replica.cc:330] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b: stopping tablet replica
I20260928 04:56:41.903594 15106 tablet_service.cc:1514] Processing DeleteTablet for tablet cac57dce140c4f9e8b0bc89dfec15c91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:56:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41126
I20260928 04:56:41.903831 15201 raft_consensus.cc:2229] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:56:41.904017 15201 raft_consensus.cc:2258] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:56:41.904637 15201 ts_tablet_manager.cc:1905] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:56:41.906440 15202 tablet_replica.cc:330] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906: stopping tablet replica
I20260928 04:56:41.906601 15202 raft_consensus.cc:2229] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:56:41.906764 15202 raft_consensus.cc:2258] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:56:41.907230 15201 ts_tablet_manager.cc:1918] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:56:41.907267 15202 ts_tablet_manager.cc:1905] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:56:41.907399 15201 log.cc:1196] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/cac57dce140c4f9e8b0bc89dfec15c91
I20260928 04:56:41.907855 15201 ts_tablet_manager.cc:1939] T cac57dce140c4f9e8b0bc89dfec15c91 P d6476329b23445a9a17e750d5a99c15b: Deleting consensus metadata
I20260928 04:56:41.908663 14700 catalog_manager.cc:4841] TS d6476329b23445a9a17e750d5a99c15b (127.0.98.2:45959): tablet cac57dce140c4f9e8b0bc89dfec15c91 (table test-table [id=b5acb62159e745c7ad680af5f908543f]) successfully deleted
I20260928 04:56:41.908973 15202 ts_tablet_manager.cc:1918] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:56:41.909053 15202 log.cc:1196] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/cac57dce140c4f9e8b0bc89dfec15c91
W20260928 04:56:41.909125 14700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41023 (error 108)
I20260928 04:56:41.909341 15202 ts_tablet_manager.cc:1939] T cac57dce140c4f9e8b0bc89dfec15c91 P 1ca74b63eda84f3998d00a8327ca9906: Deleting consensus metadata
W20260928 04:56:41.909636 15178 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:56:41.909490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41023 (local address 127.0.0.1:40222)
0928 04:56:41.909544 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:41.909587 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41023: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 04:56:41.910073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 14907
W20260928 04:56:41.909838 14700 catalog_manager.cc:4584] TS 945a0d946b324194876eeb161ac802f7 (127.0.98.1:41023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cac57dce140c4f9e8b0bc89dfec15c91: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41023: connect: Connection reset by peer (error 104)
I20260928 04:56:41.910393 14700 catalog_manager.cc:4841] TS 1ca74b63eda84f3998d00a8327ca9906 (127.0.98.3:37699): tablet cac57dce140c4f9e8b0bc89dfec15c91 (table test-table [id=b5acb62159e745c7ad680af5f908543f]) successfully deleted
I20260928 04:56:41.917083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 15043
I20260928 04:56:41.921411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 14681
I20260928 04:56:41.927120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42115
--webserver_interface=127.0.98.62
--webserver_port=43279
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:56:42.012817 15204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:42.013047 15204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:42.013073 15204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:42.014256 15204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:56:42.014299 15204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:42.014313 15204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:42.014325 15204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:56:42.014366 15204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:56:42.015877 15204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:42.016145 15204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:42.016399 15204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:42.021346 15209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:42.021333 15212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:42.021333 15210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:42.021366 15204 server_base.cc:1034] running on GCE node
I20260928 04:56:42.021816 15204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:42.022118 15204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:42.023309 15204 hybrid_clock.cc:648] HybridClock initialized: now 1790571402023303 us; error 36 us; skew 500 ppm
Sep 28 04:56:42 dist-test-slave-m39l krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:42.026392 15204 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:56:42.027099 15204 webserver.cc:460] Webserver started at http://127.0.98.62:43279/ using document root <none> and password file <none>
I20260928 04:56:42.027427 15204 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:42.027513 15204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:42.029523 15204 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:56:42.030460 15219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:42.030711 15204 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:42.030810 15204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "db67c836b4064d32b79ddd9a2437ca09"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "39789982af71f61f38f2a6c1a8f22bfb"
server_key_iv: "7ea2e28ee845080243aa99e97ec07296"
server_key_version: "encryptionkey@0"
I20260928 04:56:42.031185 15204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:42.044507 15204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:42.046721 15204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:42.046875 15204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:42.050834 15204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42115
I20260928 04:56:42.050873 15271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42115 every 8 connection(s)
I20260928 04:56:42.051280 15204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:56:42.052367 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 15204
I20260928 04:56:42.052672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41023
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42115
--webserver_port=33569
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:56:42.053364 15272 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:56:42.054383 15272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09: Bootstrap starting.
I20260928 04:56:42.062999 15272 log.cc:824] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:42.065301 15272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:56:42.065639 15272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09: Bootstrap complete.
I20260928 04:56:42.068074 15272 raft_consensus.cc:348] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db67c836b4064d32b79ddd9a2437ca09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260928 04:56:42.068337 15272 raft_consensus.cc:729] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: db67c836b4064d32b79ddd9a2437ca09, State: Initialized, Role: FOLLOWER
I20260928 04:56:42.068544 15272 consensus_queue.cc:260] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db67c836b4064d32b79ddd9a2437ca09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260928 04:56:42.068626 15272 raft_consensus.cc:388] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:56:42.068688 15272 raft_consensus.cc:482] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:56:42.068743 15272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 2 FOLLOWER]: Advancing to term 3
I20260928 04:56:42.069806 15272 raft_consensus.cc:504] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db67c836b4064d32b79ddd9a2437ca09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260928 04:56:42.069933 15272 leader_election.cc:302] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db67c836b4064d32b79ddd9a2437ca09; no voters:
I20260928 04:56:42.070174 15272 leader_election.cc:288] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 04:56:42.070212 15276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 04:56:42.070340 15276 raft_consensus.cc:686] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [term 3 LEADER]: Becoming Leader. State: Replica: db67c836b4064d32b79ddd9a2437ca09, State: Running, Role: LEADER
I20260928 04:56:42.070431 15276 consensus_queue.cc:237] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db67c836b4064d32b79ddd9a2437ca09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260928 04:56:42.070480 15272 sys_catalog.cc:564] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:56:42.070838 15277 sys_catalog.cc:455] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "db67c836b4064d32b79ddd9a2437ca09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db67c836b4064d32b79ddd9a2437ca09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } } }
I20260928 04:56:42.070926 15277 sys_catalog.cc:458] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [sys.catalog]: This master's current role is: LEADER
I20260928 04:56:42.071180 15278 sys_catalog.cc:455] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db67c836b4064d32b79ddd9a2437ca09. Latest consensus state: current_term: 3 leader_uuid: "db67c836b4064d32b79ddd9a2437ca09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db67c836b4064d32b79ddd9a2437ca09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } } }
I20260928 04:56:42.071297 15282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:56:42.071303 15278 sys_catalog.cc:458] T 00000000000000000000000000000000 P db67c836b4064d32b79ddd9a2437ca09 [sys.catalog]: This master's current role is: LEADER
I20260928 04:56:42.072273 15282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:56:42.072402 15282 catalog_manager.cc:1259] Loaded cluster ID: 61c23d17247a4d248c8a3627177a923c
I20260928 04:56:42.072628 15282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 04:56:42.072844 15282 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe1db7359ad google::LogMessage::Fail() at ??:0
@ 0x7fe1db73888a google::LogMessage::SendToLog() at ??:0
@ 0x7fe1db7354de google::LogMessage::Flush() at ??:0
@ 0x7fe1db736a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe1e36560f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe1e365f8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe1dd6030d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe1dd5f8954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe1de45c6db start_thread at ??:0
@ 0x7fe1da23361f clone at ??:0
W20260928 04:56:42.151921 15274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:42.152151 15274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:42.152184 15274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:42.153425 15274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:42.153477 15274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:42.153505 15274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:56:42.155117 15274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33569
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:42.155390 15274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:42.155668 15274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:42.160997 15309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:42.161206 15274 server_base.cc:1034] running on GCE node
W20260928 04:56:42.160997 15311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:42.161028 15308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:42.161553 15274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:42.161836 15274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:42.162994 15274 hybrid_clock.cc:648] HybridClock initialized: now 1790571402162967 us; error 44 us; skew 500 ppm
Sep 28 04:56:42 dist-test-slave-m39l krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:42.165743 15274 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:56:42.166303 15274 webserver.cc:460] Webserver started at http://127.0.98.1:33569/ using document root <none> and password file <none>
I20260928 04:56:42.166613 15274 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:42.166690 15274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:42.168455 15274 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:56:42.169191 15318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:42.169401 15274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:56:42.169499 15274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "945a0d946b324194876eeb161ac802f7"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "7df215aeea36046694f484d3e169fd69"
server_key_iv: "3f57299522ccb91183d627b1cb0d1321"
server_key_version: "encryptionkey@0"
I20260928 04:56:42.170012 15274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:42.179425 15274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:42.182212 15274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:42.182394 15274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:42.182750 15274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:56:42.183317 15325 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 04:56:42.184943 15274 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 04:56:42.185007 15274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 04:56:42.185042 15274 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 04:56:42.185976 15274 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 04:56:42.186025 15274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 04:56:42.186124 15325 tablet_bootstrap.cc:492] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7: Bootstrap starting.
I20260928 04:56:42.193835 15274 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41023
I20260928 04:56:42.194052 15432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41023 every 8 connection(s)
I20260928 04:56:42.194332 15274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:56:42.197261 15325 log.cc:824] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:42.198325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 15274
I20260928 04:56:42.198654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45959
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42115
--webserver_port=38791
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:56:42.200265 15325 tablet_bootstrap.cc:492] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:56:42.200574 15325 tablet_bootstrap.cc:492] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7: Bootstrap complete.
I20260928 04:56:42.201547 15325 ts_tablet_manager.cc:1397] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.009s
I20260928 04:56:42.202735 15325 raft_consensus.cc:348] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } }
I20260928 04:56:42.203007 15325 raft_consensus.cc:729] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 945a0d946b324194876eeb161ac802f7, State: Initialized, Role: FOLLOWER
I20260928 04:56:42.203198 15325 consensus_queue.cc:260] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } }
I20260928 04:56:42.203516 15325 ts_tablet_manager.cc:1428] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260928 04:56:42.285054 15435 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:56:42.195148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:49513)
0928 04:56:42.195358 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:42.195412 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:42.196299 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:42.284779 (+ 88480us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":98,"threads_started":1}
W20260928 04:56:42.285362 15433 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42115 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42115: Client connection negotiation failed: client connection to 127.0.98.62:42115: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 04:56:42.285759 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:42.285613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:59143)
0928 04:56:42.285680 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:42.285733 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 04:56:42.286165 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:42.286072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:58485)
0928 04:56:42.286123 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:42.286149 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 04:56:42.286322 15433 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:56:42.287482 15437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:42.287690 15437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:42.287719 15437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:42.288941 15437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:42.288977 15437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:42.289006 15437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:56:42.290563 15437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38791
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:42.290776 15437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:42.290969 15437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:42.296273 15443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:42.296350 15446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:42.296262 15444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:42.296586 15437 server_base.cc:1034] running on GCE node
I20260928 04:56:42.296931 15437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:42.297201 15437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:42.298412 15437 hybrid_clock.cc:648] HybridClock initialized: now 1790571402298388 us; error 48 us; skew 500 ppm
Sep 28 04:56:42 dist-test-slave-m39l krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:42.301152 15437 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:56:42.301668 15437 webserver.cc:460] Webserver started at http://127.0.98.2:38791/ using document root <none> and password file <none>
I20260928 04:56:42.301945 15437 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:42.302021 15437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:42.303541 15437 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:42.304311 15453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:42.304653 15437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:42.304778 15437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "d6476329b23445a9a17e750d5a99c15b"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "b206a72f3e4bb59fa2f4b1758232d597"
server_key_iv: "84e78ce9ff5f9e994773de53fae460c9"
server_key_version: "encryptionkey@0"
I20260928 04:56:42.305217 15437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:42.327059 15437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:42.329510 15437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:42.329710 15437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:42.329993 15437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:56:42.330372 15437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:56:42.330435 15437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:42.330519 15437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:56:42.330557 15437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:42.337448 15437 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45959
I20260928 04:56:42.337550 15566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45959 every 8 connection(s)
I20260928 04:56:42.337929 15437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:56:42.339267 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:42.338862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:36415)
0928 04:56:42.339039 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:42.339118 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
W20260928 04:56:42.339732 15567 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42115 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42115: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
I20260928 04:56:42.340030 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:42.339920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:36067)
0928 04:56:42.339985 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:42.340013 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 04:56:42.340462 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:42.340303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:41133)
0928 04:56:42.340401 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:42.340440 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260928 04:56:42.340761 15567 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 04:56:42.347038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 15437
I20260928 04:56:42.347268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37699
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42115
--webserver_port=42907
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:56:42.430217 15570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:42.430492 15570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:42.430572 15570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:42.431849 15570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:42.431939 15570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:42.431988 15570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:56:42.433619 15570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42907
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:42.433950 15570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:42.434223 15570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:42.439667 15576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:42.439690 15575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:42.439690 15578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:42.440044 15570 server_base.cc:1034] running on GCE node
I20260928 04:56:42.440284 15570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:42.440565 15570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:42.441767 15570 hybrid_clock.cc:648] HybridClock initialized: now 1790571402441749 us; error 30 us; skew 500 ppm
Sep 28 04:56:42 dist-test-slave-m39l krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:42.444569 15570 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:56:42.444990 15570 webserver.cc:460] Webserver started at http://127.0.98.3:42907/ using document root <none> and password file <none>
I20260928 04:56:42.445253 15570 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:42.445326 15570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:42.446801 15570 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:42.447407 15585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:42.447644 15570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:42.447752 15570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "1ca74b63eda84f3998d00a8327ca9906"
format_stamp: "Formatted at 2026-09-28 04:56:40 on dist-test-slave-m39l"
server_key: "cb02d9ea5fc0f70f99fe631afe0500ed"
server_key_iv: "af47ce2a2d714e9f8e7db995f289da0f"
server_key_version: "encryptionkey@0"
I20260928 04:56:42.448112 15570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:42.478452 15570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:42.481093 15570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:42.481300 15570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:42.481632 15570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:56:42.481986 15570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:56:42.482049 15570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:42.482096 15570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:56:42.482121 15570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:42.488392 15570 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37699
I20260928 04:56:42.488436 15698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37699 every 8 connection(s)
I20260928 04:56:42.488852 15570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:56:42.489979 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:42.489520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:49537)
0928 04:56:42.489719 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:42.489825 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":69,"threads_started":1}
W20260928 04:56:42.490396 15699 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42115 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42115: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
I20260928 04:56:42.490736 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:42.490618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:35899)
0928 04:56:42.490678 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:42.490710 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 04:56:42.491003 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:42.490954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:56211)
0928 04:56:42.490975 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:42.490992 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 04:56:42.491132 15699 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 04:56:42.492626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 15570
I20260928 04:56:43.287046 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:43.286795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:51199)
0928 04:56:43.286929 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:43.286974 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260928 04:56:43.341427 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:43.341157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:48867)
0928 04:56:43.341300 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:43.341351 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
I20260928 04:56:43.446000 15705 raft_consensus.cc:482] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:56:43.446185 15705 raft_consensus.cc:504] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } }
I20260928 04:56:43.446666 15705 leader_election.cc:288] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ca74b63eda84f3998d00a8327ca9906 (127.0.98.3:37699), d6476329b23445a9a17e750d5a99c15b (127.0.98.2:45959)
Sep 28 04:56:43 dist-test-slave-m39l krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 28 04:56:43 dist-test-slave-m39l krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260928 04:56:43.458632 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:43.447060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38661 (local address 127.0.98.3:37699)
0928 04:56:43.447250 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:43.447264 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:43.447290 (+ 26us) server_negotiation.cc:407] Connection header received
0928 04:56:43.447764 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:43.447767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:43.447824 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:43.447985 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:43.448818 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:43.449720 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:43.450843 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:43.451328 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:43.451516 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:43.454656 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:43.454659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:43.454660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:43.454672 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:43.456607 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:43.457156 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:43.457159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:43.457159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:43.457287 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:43.457610 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:43.457611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:43.457612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:43.457879 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:43.457985 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:43.458294 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 04:56:43.458445 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":69,"threads_started":1}
I20260928 04:56:43.458729 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:43.447197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42045 (local address 127.0.98.2:45959)
0928 04:56:43.447332 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:43.447344 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:43.447387 (+ 43us) server_negotiation.cc:407] Connection header received
0928 04:56:43.447844 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:43.447847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:43.447886 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:43.448041 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:43.448822 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:43.449702 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:43.450863 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:43.451371 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:43.451488 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:43.454274 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:43.454279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:43.454279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:43.454293 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:43.456428 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:43.457065 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:43.457069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:43.457070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:43.457204 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:43.457585 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:43.457587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:43.457587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:43.457889 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:43.458026 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:43.458303 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 04:56:43.458444 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":53,"threads_started":1}
I20260928 04:56:43.458730 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:43.447076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45959 (local address 127.0.98.1:42045)
0928 04:56:43.447206 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:43.447217 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:43.447298 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:43.448049 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:43.448050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:43.448465 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:43.448677 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:43.448684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:43.449827 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:43.449828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:43.450774 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:43.450777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:43.451460 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:43.451461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:43.451595 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:43.452336 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:43.452340 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:43.454091 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:43.456537 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:43.456546 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:43.456549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:43.456948 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:43.457280 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:43.457281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:43.457282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:43.457442 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:43.458012 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:43.458013 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:43.458112 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 04:56:43.458238 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 04:56:43.458585 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":179,"thread_start_us":67,"threads_started":1}
I20260928 04:56:43.458758 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:43.446898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37699 (local address 127.0.98.1:38661)
0928 04:56:43.446953 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:43.446967 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:43.447027 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:43.448033 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:43.448036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:43.448464 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:43.448676 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:43.448684 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:43.449840 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:43.449841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:43.450742 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:43.450745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:43.451474 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:43.451475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:43.451595 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:43.452294 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:43.452303 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:43.454471 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:43.456726 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:43.456727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:43.456728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:43.457058 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:43.457368 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:43.457369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:43.457369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:43.457453 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:43.457956 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:43.457963 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:43.458113 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 04:56:43.458238 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 04:56:43.458457 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":45}
W20260928 04:56:43.459762 15321 leader_election.cc:341] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6476329b23445a9a17e750d5a99c15b (127.0.98.2:45959): Not found: Tablet not found: cac57dce140c4f9e8b0bc89dfec15c91
W20260928 04:56:43.460220 15321 leader_election.cc:341] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ca74b63eda84f3998d00a8327ca9906 (127.0.98.3:37699): Not found: Tablet not found: cac57dce140c4f9e8b0bc89dfec15c91
I20260928 04:56:43.460311 15321 leader_election.cc:302] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 945a0d946b324194876eeb161ac802f7; no voters: 1ca74b63eda84f3998d00a8327ca9906, d6476329b23445a9a17e750d5a99c15b
I20260928 04:56:43.460501 15705 raft_consensus.cc:2726] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:56:43.491863 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:43.491614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:59211)
0928 04:56:43.491761 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:43.491818 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260928 04:56:44.287922 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:44.287688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:46389)
0928 04:56:44.287830 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:44.287871 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260928 04:56:44.342293 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:44.342039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:47515)
0928 04:56:44.342206 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:44.342249 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
I20260928 04:56:44.492699 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:44.492467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:49523)
0928 04:56:44.492608 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:44.492654 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260928 04:56:45.288895 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:45.288623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:56851)
0928 04:56:45.288800 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:45.288847 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":81,"threads_started":1}
I20260928 04:56:45.343273 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:45.342995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:51483)
0928 04:56:45.343172 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:45.343223 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":70,"threads_started":1}
I20260928 04:56:45.493635 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:45.493387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:44435)
0928 04:56:45.493535 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:45.493588 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260928 04:56:45.689678 15716 raft_consensus.cc:482] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:56:45.689826 15716 raft_consensus.cc:504] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } }
I20260928 04:56:45.690024 15716 leader_election.cc:288] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ca74b63eda84f3998d00a8327ca9906 (127.0.98.3:37699), d6476329b23445a9a17e750d5a99c15b (127.0.98.2:45959)
W20260928 04:56:45.690683 15321 leader_election.cc:341] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ca74b63eda84f3998d00a8327ca9906 (127.0.98.3:37699): Not found: Tablet not found: cac57dce140c4f9e8b0bc89dfec15c91
W20260928 04:56:45.690754 15321 leader_election.cc:341] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6476329b23445a9a17e750d5a99c15b (127.0.98.2:45959): Not found: Tablet not found: cac57dce140c4f9e8b0bc89dfec15c91
I20260928 04:56:45.690771 15321 leader_election.cc:302] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 945a0d946b324194876eeb161ac802f7; no voters: 1ca74b63eda84f3998d00a8327ca9906, d6476329b23445a9a17e750d5a99c15b
I20260928 04:56:45.690907 15716 raft_consensus.cc:2726] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:56:46.289911 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:46.289631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:36613)
0928 04:56:46.289811 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:46.289860 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
I20260928 04:56:46.344127 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:46.343880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:50259)
0928 04:56:46.344036 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:46.344083 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":65,"threads_started":1}
I20260928 04:56:46.494457 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:46.494203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:54213)
0928 04:56:46.494362 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:46.494409 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260928 04:56:47.290846 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:47.290628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:59425)
0928 04:56:47.290756 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:47.290800 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260928 04:56:47.345089 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:47.344729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:51019)
0928 04:56:47.344966 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:47.345020 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":89,"threads_started":1}
I20260928 04:56:47.495324 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:47.495087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:39935)
0928 04:56:47.495226 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:47.495278 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":66,"threads_started":1}
I20260928 04:56:47.597116 15723 raft_consensus.cc:482] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:56:47.597216 15723 raft_consensus.cc:504] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } }
I20260928 04:56:47.597419 15723 leader_election.cc:288] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ca74b63eda84f3998d00a8327ca9906 (127.0.98.3:37699), d6476329b23445a9a17e750d5a99c15b (127.0.98.2:45959)
W20260928 04:56:47.597915 15321 leader_election.cc:341] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6476329b23445a9a17e750d5a99c15b (127.0.98.2:45959): Not found: Tablet not found: cac57dce140c4f9e8b0bc89dfec15c91
W20260928 04:56:47.598006 15321 leader_election.cc:341] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ca74b63eda84f3998d00a8327ca9906 (127.0.98.3:37699): Not found: Tablet not found: cac57dce140c4f9e8b0bc89dfec15c91
I20260928 04:56:47.598040 15321 leader_election.cc:302] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 945a0d946b324194876eeb161ac802f7; no voters: 1ca74b63eda84f3998d00a8327ca9906, d6476329b23445a9a17e750d5a99c15b
I20260928 04:56:47.598132 15723 raft_consensus.cc:2726] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:56:48.291728 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:48.291471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:41849)
0928 04:56:48.291632 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:48.291681 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":64,"threads_started":1}
I20260928 04:56:48.345928 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:48.345663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:47657)
0928 04:56:48.345817 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:48.345863 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260928 04:56:48.496115 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:48.495890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:54659)
0928 04:56:48.496022 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:48.496067 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260928 04:56:49.292649 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:49.292347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:55921)
0928 04:56:49.292554 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:49.292600 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":63,"threads_started":1}
I20260928 04:56:49.346856 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:49.346600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:46171)
0928 04:56:49.346761 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:49.346810 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260928 04:56:49.496979 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:49.496663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:51571)
0928 04:56:49.496805 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:49.496914 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260928 04:56:50.293478 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:50.293225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:56019)
0928 04:56:50.293376 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:50.293419 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260928 04:56:50.347672 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:50.347431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:48477)
0928 04:56:50.347576 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:50.347623 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260928 04:56:50.497910 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:50.497635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:44625)
0928 04:56:50.497801 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:50.497847 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":63,"threads_started":1}
I20260928 04:56:50.743903 15733 raft_consensus.cc:482] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:56:50.744019 15733 raft_consensus.cc:504] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } }
I20260928 04:56:50.744257 15733 leader_election.cc:288] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ca74b63eda84f3998d00a8327ca9906 (127.0.98.3:37699), d6476329b23445a9a17e750d5a99c15b (127.0.98.2:45959)
W20260928 04:56:50.744735 15321 leader_election.cc:341] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ca74b63eda84f3998d00a8327ca9906 (127.0.98.3:37699): Not found: Tablet not found: cac57dce140c4f9e8b0bc89dfec15c91
W20260928 04:56:50.744858 15321 leader_election.cc:341] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6476329b23445a9a17e750d5a99c15b (127.0.98.2:45959): Not found: Tablet not found: cac57dce140c4f9e8b0bc89dfec15c91
I20260928 04:56:50.744895 15321 leader_election.cc:302] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 945a0d946b324194876eeb161ac802f7; no voters: 1ca74b63eda84f3998d00a8327ca9906, d6476329b23445a9a17e750d5a99c15b
I20260928 04:56:50.744969 15733 raft_consensus.cc:2726] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:56:51.294394 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:51.294089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:40213)
0928 04:56:51.294283 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:51.294340 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260928 04:56:51.348667 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:51.348262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:47103)
0928 04:56:51.348530 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:51.348580 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":81,"threads_started":1}
I20260928 04:56:51.498852 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:51.498592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:33331)
0928 04:56:51.498765 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:51.498806 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":76,"threads_started":1}
I20260928 04:56:52.295373 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:52.295041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:36267)
0928 04:56:52.295271 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:52.295317 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":133,"threads_started":1}
I20260928 04:56:52.349641 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:52.349290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:57255)
0928 04:56:52.349478 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:52.349595 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":78,"threads_started":1}
I20260928 04:56:52.499843 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:52.499564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:53461)
0928 04:56:52.499732 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:52.499781 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":66,"threads_started":1}
I20260928 04:56:53.296228 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:53.295978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:37975)
0928 04:56:53.296133 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:53.296181 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":62,"threads_started":1}
I20260928 04:56:53.350668 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:53.350275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:34185)
0928 04:56:53.350571 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:53.350620 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":137,"threads_started":1}
I20260928 04:56:53.500802 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:53.500496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:50225)
0928 04:56:53.500704 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:53.500753 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
I20260928 04:56:54.297263 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:54.296923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:34765)
0928 04:56:54.297143 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:54.297215 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":85,"threads_started":1}
I20260928 04:56:54.351578 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:54.351222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:51595)
0928 04:56:54.351412 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:54.351473 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":87,"threads_started":1}
I20260928 04:56:54.501741 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:54.501443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:46383)
0928 04:56:54.501644 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:54.501692 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
I20260928 04:56:54.618752 15746 raft_consensus.cc:482] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:56:54.618868 15746 raft_consensus.cc:504] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca74b63eda84f3998d00a8327ca9906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "d6476329b23445a9a17e750d5a99c15b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45959 } } peers { permanent_uuid: "945a0d946b324194876eeb161ac802f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41023 } }
I20260928 04:56:54.619202 15746 leader_election.cc:288] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ca74b63eda84f3998d00a8327ca9906 (127.0.98.3:37699), d6476329b23445a9a17e750d5a99c15b (127.0.98.2:45959)
W20260928 04:56:54.619717 15321 leader_election.cc:341] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ca74b63eda84f3998d00a8327ca9906 (127.0.98.3:37699): Not found: Tablet not found: cac57dce140c4f9e8b0bc89dfec15c91
W20260928 04:56:54.619822 15321 leader_election.cc:341] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6476329b23445a9a17e750d5a99c15b (127.0.98.2:45959): Not found: Tablet not found: cac57dce140c4f9e8b0bc89dfec15c91
I20260928 04:56:54.619855 15321 leader_election.cc:302] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 945a0d946b324194876eeb161ac802f7; no voters: 1ca74b63eda84f3998d00a8327ca9906, d6476329b23445a9a17e750d5a99c15b
I20260928 04:56:54.619957 15746 raft_consensus.cc:2726] T cac57dce140c4f9e8b0bc89dfec15c91 P 945a0d946b324194876eeb161ac802f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:56:55.298177 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:55.297922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:56359)
0928 04:56:55.298089 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:55.298133 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
I20260928 04:56:55.352342 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:55.352094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:50951)
0928 04:56:55.352248 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:55.352292 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260928 04:56:55.502483 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:55.502248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:55323)
0928 04:56:55.502384 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:55.502431 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260928 04:56:56.299161 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:56.298864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:36979)
0928 04:56:56.299060 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:56.299111 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":90,"threads_started":1}
I20260928 04:56:56.353219 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:56.352949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:48233)
0928 04:56:56.353107 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:56.353161 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
I20260928 04:56:56.503290 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:56.503059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:59271)
0928 04:56:56.503194 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:56.503243 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260928 04:56:57.300127 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:57.299835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:58029)
0928 04:56:57.300026 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:57.300078 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
I20260928 04:56:57.354279 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:57.354029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:56967)
0928 04:56:57.354177 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:57.354224 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":61,"threads_started":1}
I20260928 04:56:57.497213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 15274
I20260928 04:56:57.502643 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 15437
I20260928 04:56:57.504141 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:57.503848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:49145)
0928 04:56:57.504036 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:57.504080 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":58,"threads_started":1}
I20260928 04:56:57.507215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 15570
2026-09-28T04:56:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18455 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:56:57 dist-test-slave-m39l krb5kdc[15758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:56:57 dist-test-slave-m39l krb5kdc[15758](info): set up 2 sockets
Sep 28 04:56:57 dist-test-slave-m39l krb5kdc[15758](info): commencing operation
krb5kdc: starting...
W20260928 04:56:58.635192 392 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:56:58 dist-test-slave-m39l krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:56:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:56:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:56:58.765842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43201
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:56:58.852499 15771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:58.852774 15771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:58.852813 15771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:58.854004 15771 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 04:56:58.854089 15771 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 04:56:58.854110 15771 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 04:56:58.854130 15771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:56:58.854161 15771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:58.854193 15771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:58.854216 15771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:56:58.854245 15771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:56:58.855863 15771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:58.856170 15771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:58.856423 15771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:56:58.862202 15771 server_base.cc:1034] running on GCE node
W20260928 04:56:58.862202 15779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:58.862457 15777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:58.862630 15776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:58.862886 15771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:58.863240 15771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:58.864404 15771 hybrid_clock.cc:648] HybridClock initialized: now 1790571418864397 us; error 52 us; skew 500 ppm
Sep 28 04:56:58 dist-test-slave-m39l krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:58.867066 15771 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:56:58.867563 15771 webserver.cc:460] Webserver started at http://127.0.98.62:39071/ using document root <none> and password file <none>
I20260928 04:56:58.867805 15771 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:58.867877 15771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:58.868018 15771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:56:58.869123 15771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e"
format_stamp: "Formatted at 2026-09-28 04:56:58 on dist-test-slave-m39l"
server_key: "3c9d3abd5d00152068d5979ded4f764d"
server_key_iv: "5a2044b69944d6842959eb24da01ed27"
server_key_version: "encryptionkey@0"
I20260928 04:56:58.869508 15771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e"
format_stamp: "Formatted at 2026-09-28 04:56:58 on dist-test-slave-m39l"
server_key: "3c9d3abd5d00152068d5979ded4f764d"
server_key_iv: "5a2044b69944d6842959eb24da01ed27"
server_key_version: "encryptionkey@0"
I20260928 04:56:58.871034 15771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:56:58.871927 15786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:58.872135 15771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:58.872257 15771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e"
format_stamp: "Formatted at 2026-09-28 04:56:58 on dist-test-slave-m39l"
server_key: "3c9d3abd5d00152068d5979ded4f764d"
server_key_iv: "5a2044b69944d6842959eb24da01ed27"
server_key_version: "encryptionkey@0"
I20260928 04:56:58.872359 15771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:58.889624 15771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:58.892204 15771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:58.892402 15771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:58.897254 15771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43201
I20260928 04:56:58.897365 15838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43201 every 8 connection(s)
I20260928 04:56:58.897748 15771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:56:58.898818 15839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:58.901351 15839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: Bootstrap starting.
I20260928 04:56:58.901543 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 15771
I20260928 04:56:58.901671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:56:58.901866 392 external_mini_cluster.cc:1428] Setting key 16b71097772a3f0a42ffbdb7c7655c67
I20260928 04:56:58.902120 15839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:58.902541 15839 log.cc:824] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:58.903368 15839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: No bootstrap required, opened a new log
I20260928 04:56:58.905925 15839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } }
I20260928 04:56:58.906112 15839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:58.906147 15839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cdfd8e2733a4f2d9afb456cdc3ed10e, State: Initialized, Role: FOLLOWER
I20260928 04:56:58.906275 15839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } }
I20260928 04:56:58.906349 15839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:56:58.906381 15839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:56:58.906423 15839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:58.907106 15839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } }
I20260928 04:56:58.907228 15839 leader_election.cc:302] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cdfd8e2733a4f2d9afb456cdc3ed10e; no voters:
I20260928 04:56:58.907429 15839 leader_election.cc:288] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:56:58.907650 15845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:56:58.907809 15839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:56:58.907814 15845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 1 LEADER]: Becoming Leader. State: Replica: 9cdfd8e2733a4f2d9afb456cdc3ed10e, State: Running, Role: LEADER
I20260928 04:56:58.908032 15845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } }
I20260928 04:56:58.908456 15846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cdfd8e2733a4f2d9afb456cdc3ed10e. Latest consensus state: current_term: 1 leader_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } } }
I20260928 04:56:58.908566 15846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [sys.catalog]: This master's current role is: LEADER
I20260928 04:56:58.908459 15844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } } }
I20260928 04:56:58.908795 15844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [sys.catalog]: This master's current role is: LEADER
I20260928 04:56:58.908933 15853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 04:56:58 dist-test-slave-m39l krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:58.909765 15853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:56:58.912698 15853 catalog_manager.cc:1347] Generated new cluster ID: a8b3f961a235402cbaa7aff44d891532
I20260928 04:56:58.912756 15853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:56:58.914424 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:58.902536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53214 (local address 127.0.98.62:43201)
0928 04:56:58.902757 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:58.902766 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:58.902814 (+ 48us) server_negotiation.cc:407] Connection header received
0928 04:56:58.903252 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:58.903259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:58.903308 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:58.903472 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:58.904314 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:58.905346 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:58.906581 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:58.907033 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:58.907123 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:58.910411 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:58.910415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:58.910416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:58.910431 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:58.912526 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:58.912980 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:58.912984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:58.912984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:58.913107 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:58.913339 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:58.913340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:58.913340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:58.913680 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:58.913849 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:58.914029 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 04:56:58.914188 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":89,"threads_started":1}
I20260928 04:56:58.962744 15853 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:56:58.963361 15853 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:56:58.970642 15853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: Generated new TSK 0
I20260928 04:56:58.970837 15853 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:56:59.002409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43201
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:56:59.096396 15865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:59.096668 15865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:59.096700 15865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:59.098228 15865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:59.098294 15865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:59.098330 15865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:56:59.100169 15865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:59.100646 15865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:59.101009 15865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:59.108742 15873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:59.108742 15871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:59.109269 15870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:59.109578 15865 server_base.cc:1034] running on GCE node
I20260928 04:56:59.109897 15865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:59.110213 15865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:59.111487 15865 hybrid_clock.cc:648] HybridClock initialized: now 1790571419111472 us; error 36 us; skew 500 ppm
Sep 28 04:56:59 dist-test-slave-m39l krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:59.115126 15865 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:56:59.115722 15865 webserver.cc:460] Webserver started at http://127.0.98.1:40287/ using document root <none> and password file <none>
I20260928 04:56:59.116000 15865 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:59.116079 15865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:59.116251 15865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:56:59.117450 15865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "f57d763bdbdf4a00b28759301947652e"
format_stamp: "Formatted at 2026-09-28 04:56:59 on dist-test-slave-m39l"
server_key: "1d43a86e45ffe46192c31ad5c21e127e"
server_key_iv: "b557847cb6f680a77b09c8b95b3d8422"
server_key_version: "encryptionkey@0"
I20260928 04:56:59.117879 15865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f57d763bdbdf4a00b28759301947652e"
format_stamp: "Formatted at 2026-09-28 04:56:59 on dist-test-slave-m39l"
server_key: "1d43a86e45ffe46192c31ad5c21e127e"
server_key_iv: "b557847cb6f680a77b09c8b95b3d8422"
server_key_version: "encryptionkey@0"
I20260928 04:56:59.119959 15865 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 04:56:59.121054 15880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:59.121464 15865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:59.121592 15865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "f57d763bdbdf4a00b28759301947652e"
format_stamp: "Formatted at 2026-09-28 04:56:59 on dist-test-slave-m39l"
server_key: "1d43a86e45ffe46192c31ad5c21e127e"
server_key_iv: "b557847cb6f680a77b09c8b95b3d8422"
server_key_version: "encryptionkey@0"
I20260928 04:56:59.121683 15865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:59.146008 15865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:59.149019 15865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:59.149247 15865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:59.149631 15865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:56:59.150027 15865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:56:59.150079 15865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:59.150110 15865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:56:59.150126 15865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:59.159413 15993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33483 every 8 connection(s)
I20260928 04:56:59.159456 15865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33483
I20260928 04:56:59.160061 15865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:56:59 dist-test-slave-m39l krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:59.169026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 15865
I20260928 04:56:59.169178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:56:59.169561 392 external_mini_cluster.cc:1428] Setting key 376982446fd5ce4bb8e930ffe8343854
I20260928 04:56:59.173260 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.161160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38207 (local address 127.0.98.62:43201)
0928 04:56:59.161231 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:59.161233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:59.161469 (+ 236us) server_negotiation.cc:407] Connection header received
0928 04:56:59.162059 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:59.162060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:59.162073 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:59.162131 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:59.163231 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.163753 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.164796 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.165025 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.165104 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:59.168073 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:59.168075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:59.168075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:59.168086 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:59.170270 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:59.171028 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:59.171032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:59.171032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:59.171202 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:59.171934 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:59.171936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:59.171937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:59.172154 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:59.172240 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:59.172814 (+ 574us) server_negotiation.cc:299] Negotiation successful
0928 04:56:59.173028 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 04:56:59.173556 15996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.161157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:38207)
0928 04:56:59.161340 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:59.161378 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:59.161906 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:59.162213 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:59.162217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:59.162897 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:59.163111 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:59.163116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.163871 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.163872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:59.164698 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:59.164700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.165115 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.165117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:59.165247 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:59.166098 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:59.166108 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:59.167950 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:59.170370 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:59.170381 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:59.170384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:59.170824 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:59.171648 (+ 824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:59.171649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:59.171649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:59.171794 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:59.172393 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:59.172401 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:59.172584 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 04:56:59.172698 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 04:56:59.173077 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260928 04:56:59.174729 15994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43201
I20260928 04:56:59.174871 15994 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:59.175098 15994 heartbeater.cc:511] Master 127.0.98.62:43201 requested a full tablet report, sending...
I20260928 04:56:59.175868 15803 ts_manager.cc:194] Registered new tserver with Master: f57d763bdbdf4a00b28759301947652e (127.0.98.1:33483)
I20260928 04:56:59.176537 15803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38207
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:56:59.209692 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43201
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:56:59.291635 15999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:59.291889 15999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:59.291930 15999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:59.293148 15999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:59.293220 15999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:59.293267 15999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:56:59.294824 15999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:59.295099 15999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:59.295342 15999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:59.300714 16007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:59.300714 16005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:59.300745 16004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:59.300743 15999 server_base.cc:1034] running on GCE node
I20260928 04:56:59.301227 15999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:59.301515 15999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:59.302690 15999 hybrid_clock.cc:648] HybridClock initialized: now 1790571419302690 us; error 43 us; skew 500 ppm
Sep 28 04:56:59 dist-test-slave-m39l krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:59.305481 15999 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:56:59.305948 15999 webserver.cc:460] Webserver started at http://127.0.98.2:34125/ using document root <none> and password file <none>
I20260928 04:56:59.306180 15999 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:59.306264 15999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:59.306414 15999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:56:59.307569 15999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "0930e5479d854889bd0ebe4650232873"
format_stamp: "Formatted at 2026-09-28 04:56:59 on dist-test-slave-m39l"
server_key: "28bd1ca9f5172e33acd9ee3df0a9ff5d"
server_key_iv: "61a7147a294539278f8421286702090c"
server_key_version: "encryptionkey@0"
I20260928 04:56:59.307979 15999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0930e5479d854889bd0ebe4650232873"
format_stamp: "Formatted at 2026-09-28 04:56:59 on dist-test-slave-m39l"
server_key: "28bd1ca9f5172e33acd9ee3df0a9ff5d"
server_key_iv: "61a7147a294539278f8421286702090c"
server_key_version: "encryptionkey@0"
I20260928 04:56:59.309725 15999 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 04:56:59.310622 16014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:59.310818 15999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:59.310957 15999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "0930e5479d854889bd0ebe4650232873"
format_stamp: "Formatted at 2026-09-28 04:56:59 on dist-test-slave-m39l"
server_key: "28bd1ca9f5172e33acd9ee3df0a9ff5d"
server_key_iv: "61a7147a294539278f8421286702090c"
server_key_version: "encryptionkey@0"
I20260928 04:56:59.311064 15999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:59.321110 15999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:59.323204 15999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:59.323371 15999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:59.323626 15999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:56:59.323984 15999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:56:59.324054 15999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:59.324120 15999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:56:59.324150 15999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:59.330361 15999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34075
I20260928 04:56:59.330410 16127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34075 every 8 connection(s)
I20260928 04:56:59.330775 15999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:56:59.334975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 15999
I20260928 04:56:59.335103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:56:59.335327 392 external_mini_cluster.cc:1428] Setting key 02973683df3d041986f3c417da83d577
Sep 28 04:56:59 dist-test-slave-m39l krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:59.342033 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.331549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46557 (local address 127.0.98.62:43201)
0928 04:56:59.331612 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:59.331614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:59.331986 (+ 372us) server_negotiation.cc:407] Connection header received
0928 04:56:59.332570 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:59.332572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:59.332585 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:59.332648 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:59.333760 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.334330 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.335936 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.336165 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.336256 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:59.338743 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:59.338745 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:59.338745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:59.338753 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:59.340369 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:59.340891 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:59.340892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:59.340893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:59.340961 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:59.341255 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:59.341256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:59.341257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:59.341391 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:59.341484 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:59.341816 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 04:56:59.341916 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:56:59.342288 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.331644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:46557)
0928 04:56:59.331825 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:59.331864 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:59.332451 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:59.332729 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:59.332733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:59.333431 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:59.333649 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:59.333654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.334448 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.334450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:59.335820 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:59.335823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.336262 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.336263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:59.336407 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:59.337007 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:59.337016 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:59.338625 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:59.340496 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:59.340506 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:59.340509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:59.340785 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:59.341038 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:59.341039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:59.341039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:59.341170 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:59.341472 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:59.341479 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:59.341647 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 04:56:59.341792 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 04:56:59.342000 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":75,"threads_started":1}
I20260928 04:56:59.342841 16128 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43201
I20260928 04:56:59.342964 16128 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 04:56:59.343180 16128 heartbeater.cc:511] Master 127.0.98.62:43201 requested a full tablet report, sending...
I20260928 04:56:59.343629 15803 ts_manager.cc:194] Registered new tserver with Master: 0930e5479d854889bd0ebe4650232873 (127.0.98.2:34075)
I20260928 04:56:59.344215 15803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:56:59.362330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43201
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:56:59.445449 16133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:56:59.445660 16133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:56:59.445683 16133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:56:59.446954 16133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:56:59.446995 16133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:56:59.447022 16133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:56:59.448451 16133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:56:59.448695 16133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:56:59.448941 16133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:56:59.454203 16141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:56:59.454203 16138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:59.454221 16133 server_base.cc:1034] running on GCE node
W20260928 04:56:59.454219 16139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:56:59.454731 16133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:56:59.454995 16133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:56:59.456144 16133 hybrid_clock.cc:648] HybridClock initialized: now 1790571419456112 us; error 54 us; skew 500 ppm
Sep 28 04:56:59 dist-test-slave-m39l krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:56:59.458884 16133 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:56:59.459335 16133 webserver.cc:460] Webserver started at http://127.0.98.3:40209/ using document root <none> and password file <none>
I20260928 04:56:59.459579 16133 fs_manager.cc:359] Metadata directory not provided
I20260928 04:56:59.459626 16133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:56:59.459750 16133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:56:59.460831 16133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "73844ccbbec74d319e8a1548799736c3"
format_stamp: "Formatted at 2026-09-28 04:56:59 on dist-test-slave-m39l"
server_key: "aa55783c70742e51ad1572a8c3762208"
server_key_iv: "ed2cae04595b783e44f399ba809d7e2b"
server_key_version: "encryptionkey@0"
I20260928 04:56:59.461191 16133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "73844ccbbec74d319e8a1548799736c3"
format_stamp: "Formatted at 2026-09-28 04:56:59 on dist-test-slave-m39l"
server_key: "aa55783c70742e51ad1572a8c3762208"
server_key_iv: "ed2cae04595b783e44f399ba809d7e2b"
server_key_version: "encryptionkey@0"
I20260928 04:56:59.462724 16133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:56:59.463546 16148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:59.463747 16133 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:56:59.463847 16133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "73844ccbbec74d319e8a1548799736c3"
format_stamp: "Formatted at 2026-09-28 04:56:59 on dist-test-slave-m39l"
server_key: "aa55783c70742e51ad1572a8c3762208"
server_key_iv: "ed2cae04595b783e44f399ba809d7e2b"
server_key_version: "encryptionkey@0"
I20260928 04:56:59.463946 16133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:56:59.472421 16133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:56:59.474545 16133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:56:59.474715 16133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:56:59.474960 16133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:56:59.475334 16133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:56:59.475381 16133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:59.475430 16133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:56:59.475454 16133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:56:59.481809 16133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42039
I20260928 04:56:59.481848 16261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42039 every 8 connection(s)
I20260928 04:56:59.482194 16133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:56:59.487437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 16133
I20260928 04:56:59.487551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:56:59.487747 392 external_mini_cluster.cc:1428] Setting key 807f52165a5e047b873f5882e95c0822
Sep 28 04:56:59 dist-test-slave-m39l krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:56:59.492666 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.482983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55781 (local address 127.0.98.62:43201)
0928 04:56:59.483045 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:59.483048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:59.483273 (+ 225us) server_negotiation.cc:407] Connection header received
0928 04:56:59.483817 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:59.483818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:59.483831 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:59.483873 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:59.484917 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.485439 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.486483 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.486734 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.486815 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:59.489650 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:59.489652 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:59.489652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:59.489659 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:59.491065 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:59.491536 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:59.491538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:59.491538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:59.491603 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:59.491905 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:59.491906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:59.491907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:59.492064 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:59.492144 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:59.492475 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 04:56:59.492570 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 04:56:59.492892 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.482950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:55781)
0928 04:56:59.483144 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:59.483182 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:59.483709 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:59.483960 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:59.483965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:59.484589 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:56:59.484794 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:59.484800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.485549 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.485550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:59.486381 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:59.486383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.486835 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.486837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:59.486977 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:59.487626 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:56:59.487643 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:56:59.489551 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:56:59.491145 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:59.491151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:59.491153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:59.491436 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:59.491683 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:56:59.491683 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:56:59.491684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:56:59.491812 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:56:59.492145 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:56:59.492153 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:56:59.492293 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 04:56:59.492388 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 04:56:59.492586 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
I20260928 04:56:59.493484 16262 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43201
I20260928 04:56:59.493569 16262 heartbeater.cc:464] Registering TS with master...
I20260928 04:56:59.493721 16262 heartbeater.cc:511] Master 127.0.98.62:43201 requested a full tablet report, sending...
I20260928 04:56:59.494143 15803 ts_manager.cc:194] Registered new tserver with Master: 73844ccbbec74d319e8a1548799736c3 (127.0.98.3:42039)
I20260928 04:56:59.494678 15803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55781
I20260928 04:56:59.499771 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:56:59.508021 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.501184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53222 (local address 127.0.98.62:43201)
0928 04:56:59.501265 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:59.501267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:59.501338 (+ 71us) server_negotiation.cc:407] Connection header received
0928 04:56:59.501375 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:59.501375 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:59.501385 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:59.501432 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:56:59.501995 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.502715 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.503499 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.503702 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.503765 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:59.505142 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:56:59.505144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:56:59.505145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:56:59.505151 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:56:59.506843 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:59.507182 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:59.507183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:59.507183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:59.507235 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:56:59.507490 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:56:59.507491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:56:59.507491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:56:59.507642 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:56:59.507719 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:59.507829 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 04:56:59.507900 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 04:56:59.510020 15803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:56:59.511932 15803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:56:59.519572 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.514790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.0.98.3:42039)
0928 04:56:59.515091 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:59.515099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:59.515174 (+ 75us) server_negotiation.cc:407] Connection header received
0928 04:56:59.515347 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:59.515350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:59.515369 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:59.515433 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:59.515915 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.517073 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.518312 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.519133 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.519215 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:59.519278 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:59.519422 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 04:56:59.519470 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260928 04:56:59.519572 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.514699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42039 (local address 127.0.0.1:59240)
0928 04:56:59.515050 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:59.515100 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:59.515217 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:59.515569 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:59.515573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:59.515576 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:59.515811 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:59.515813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.517184 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.517185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:59.518210 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:59.518213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.519234 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.519235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:59.519303 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:59.519316 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 04:56:59.519386 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:56:59.519439 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":7,"thread_start_us":68,"threads_started":1}
I20260928 04:56:59.520188 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.514700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34075 (local address 127.0.0.1:42390)
0928 04:56:59.515143 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:59.515152 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:59.515217 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:59.515607 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:59.515608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:59.515609 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:59.515678 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:59.515680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.516932 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.516933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:59.518561 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:59.518563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.519525 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.519526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:59.519601 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:59.519606 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:56:59.520059 (+ 453us) client_negotiation.cc:240] Negotiation successful
0928 04:56:59.520117 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":54,"thread_start_us":124,"threads_started":1}
I20260928 04:56:59.520627 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.514924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42390 (local address 127.0.98.2:34075)
0928 04:56:59.515388 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:59.515395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:59.515419 (+ 24us) server_negotiation.cc:407] Connection header received
0928 04:56:59.515485 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:59.515487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:59.515504 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:59.515643 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:59.515797 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.516792 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.518635 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.519386 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.519463 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:59.519518 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:59.520494 (+ 976us) server_negotiation.cc:299] Negotiation successful
0928 04:56:59.520543 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":86,"threads_started":1}
I20260928 04:56:59.520504 16196 tablet_service.cc:1467] Processing CreateTablet for tablet 46e87e1e16154b529dcfde5cf38eb928 (DEFAULT_TABLE table=test-table [id=460be993e0924ad78d4ff5ecf4299f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:56:59.520670 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.514700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33483 (local address 127.0.0.1:40334)
0928 04:56:59.515195 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:59.515206 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:59.515273 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:59.515630 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:59.515631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:59.515633 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:59.515688 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:59.515689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.517055 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.517056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:59.518547 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:59.518550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.520414 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.520415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:59.520485 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:59.520489 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:56:59.520562 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 04:56:59.520606 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":277,"thread_start_us":75,"threads_started":1}
I20260928 04:56:59.520924 16196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e87e1e16154b529dcfde5cf38eb928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:59.521082 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.515017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.0.98.1:33483)
0928 04:56:59.515166 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:59.515171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:59.515276 (+ 105us) server_negotiation.cc:407] Connection header received
0928 04:56:59.515422 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:59.515424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:59.515437 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:59.515538 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:59.515771 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.516945 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.519560 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.520316 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.520392 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:59.520447 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:59.520577 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 04:56:59.520618 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260928 04:56:59.521699 15928 tablet_service.cc:1467] Processing CreateTablet for tablet 46e87e1e16154b529dcfde5cf38eb928 (DEFAULT_TABLE table=test-table [id=460be993e0924ad78d4ff5ecf4299f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:56:59.522014 15928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e87e1e16154b529dcfde5cf38eb928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:59.524284 16279 tablet_bootstrap.cc:492] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3: Bootstrap starting.
I20260928 04:56:59.525033 16062 tablet_service.cc:1467] Processing CreateTablet for tablet 46e87e1e16154b529dcfde5cf38eb928 (DEFAULT_TABLE table=test-table [id=460be993e0924ad78d4ff5ecf4299f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:56:59.525281 16279 tablet_bootstrap.cc:654] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:59.525398 16062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e87e1e16154b529dcfde5cf38eb928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:56:59.525548 16280 tablet_bootstrap.cc:492] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e: Bootstrap starting.
I20260928 04:56:59.526188 16279 log.cc:824] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:59.526417 16280 tablet_bootstrap.cc:654] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:59.527228 16279 tablet_bootstrap.cc:492] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3: No bootstrap required, opened a new log
I20260928 04:56:59.527302 16279 ts_tablet_manager.cc:1397] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 04:56:59.527232 16280 log.cc:824] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:59.528427 16280 tablet_bootstrap.cc:492] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e: No bootstrap required, opened a new log
I20260928 04:56:59.528595 16280 ts_tablet_manager.cc:1397] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 04:56:59.528924 16279 raft_consensus.cc:348] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } } peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } }
I20260928 04:56:59.529083 16279 raft_consensus.cc:374] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:59.529114 16279 raft_consensus.cc:729] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73844ccbbec74d319e8a1548799736c3, State: Initialized, Role: FOLLOWER
I20260928 04:56:59.529206 16279 consensus_queue.cc:260] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } } peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } }
I20260928 04:56:59.529515 16279 ts_tablet_manager.cc:1428] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:56:59.529515 16262 heartbeater.cc:503] Master 127.0.98.62:43201 was elected leader, sending a full tablet report...
I20260928 04:56:59.529835 16284 tablet_bootstrap.cc:492] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: Bootstrap starting.
I20260928 04:56:59.530591 16284 tablet_bootstrap.cc:654] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: Neither blocks nor log segments found. Creating new log.
I20260928 04:56:59.531201 16284 log.cc:824] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: Log is configured to *not* fsync() on all Append() calls
I20260928 04:56:59.531574 16280 raft_consensus.cc:348] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } } peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } }
I20260928 04:56:59.531767 16280 raft_consensus.cc:374] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:59.531809 16280 raft_consensus.cc:729] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f57d763bdbdf4a00b28759301947652e, State: Initialized, Role: FOLLOWER
I20260928 04:56:59.531929 16280 consensus_queue.cc:260] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } } peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } }
I20260928 04:56:59.532084 16284 tablet_bootstrap.cc:492] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: No bootstrap required, opened a new log
I20260928 04:56:59.532159 16284 ts_tablet_manager.cc:1397] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:56:59.532266 16280 ts_tablet_manager.cc:1428] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 04:56:59.532286 15994 heartbeater.cc:503] Master 127.0.98.62:43201 was elected leader, sending a full tablet report...
I20260928 04:56:59.533986 16284 raft_consensus.cc:348] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } } peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } }
I20260928 04:56:59.534145 16284 raft_consensus.cc:374] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:56:59.534170 16284 raft_consensus.cc:729] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0930e5479d854889bd0ebe4650232873, State: Initialized, Role: FOLLOWER
I20260928 04:56:59.534303 16284 consensus_queue.cc:260] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } } peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } }
I20260928 04:56:59.534644 16284 ts_tablet_manager.cc:1428] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 04:56:59.534749 16128 heartbeater.cc:503] Master 127.0.98.62:43201 was elected leader, sending a full tablet report...
W20260928 04:56:59.581360 16129 tablet.cc:2369] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:56:59.661154 15995 tablet.cc:2369] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:56:59.732820 16263 tablet.cc:2369] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:56:59.782485 16283 raft_consensus.cc:482] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:56:59.782694 16283 raft_consensus.cc:504] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } } peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } }
I20260928 04:56:59.783162 16283 leader_election.cc:288] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f57d763bdbdf4a00b28759301947652e (127.0.98.1:33483), 0930e5479d854889bd0ebe4650232873 (127.0.98.2:34075)
I20260928 04:56:59.786144 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.783383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33483 (local address 127.0.98.3:59351)
0928 04:56:59.783445 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:59.783459 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:59.783513 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:59.783679 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:59.783680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:59.783682 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:59.783743 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:59.783745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.784450 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.784451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:59.785252 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:59.785255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.785848 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.785849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:59.785936 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:59.785940 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:56:59.785991 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 04:56:59.786044 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 04:56:59.786327 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.783478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59351 (local address 127.0.98.1:33483)
0928 04:56:59.783525 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:59.783527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:59.783540 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:56:59.783572 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:59.783573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:59.783586 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:59.783649 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:59.783856 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.784356 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.785370 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.785750 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.785848 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:59.785867 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:59.786227 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 04:56:59.786271 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:56:59.786728 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.783433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34075 (local address 127.0.98.3:50575)
0928 04:56:59.783626 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:59.783635 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:56:59.783680 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:56:59.783829 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:56:59.783830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:56:59.783832 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:56:59.783916 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:59.783918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.784554 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.784555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:59.785290 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:56:59.785292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.786313 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.786314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:56:59.786511 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:59.786516 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:56:59.786573 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 04:56:59.786624 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":1,"thread_start_us":119,"threads_started":1}
I20260928 04:56:59.786769 15948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e87e1e16154b529dcfde5cf38eb928" candidate_uuid: "73844ccbbec74d319e8a1548799736c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f57d763bdbdf4a00b28759301947652e" is_pre_election: true
I20260928 04:56:59.786983 15948 raft_consensus.cc:2454] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73844ccbbec74d319e8a1548799736c3 in term 0.
I20260928 04:56:59.786995 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.783519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50575 (local address 127.0.98.2:34075)
0928 04:56:59.783568 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:59.783570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:59.783704 (+ 134us) server_negotiation.cc:407] Connection header received
0928 04:56:59.783735 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:59.783736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:59.783748 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:59.783827 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:56:59.783999 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.784454 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.785407 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.786071 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.786152 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:59.786164 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:59.786856 (+ 692us) server_negotiation.cc:299] Negotiation successful
0928 04:56:59.786910 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:56:59.787257 16150 leader_election.cc:302] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73844ccbbec74d319e8a1548799736c3, f57d763bdbdf4a00b28759301947652e; no voters:
I20260928 04:56:59.787397 16283 raft_consensus.cc:2781] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:56:59.787456 16283 raft_consensus.cc:482] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:56:59.787483 16283 raft_consensus.cc:3037] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:59.788995 16283 raft_consensus.cc:504] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } } peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } }
I20260928 04:56:59.789458 16283 leader_election.cc:288] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [CANDIDATE]: Term 1 election: Requested vote from peers f57d763bdbdf4a00b28759301947652e (127.0.98.1:33483), 0930e5479d854889bd0ebe4650232873 (127.0.98.2:34075)
I20260928 04:56:59.789871 15948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e87e1e16154b529dcfde5cf38eb928" candidate_uuid: "73844ccbbec74d319e8a1548799736c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f57d763bdbdf4a00b28759301947652e"
I20260928 04:56:59.789978 15948 raft_consensus.cc:3037] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:59.791088 15948 raft_consensus.cc:2454] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73844ccbbec74d319e8a1548799736c3 in term 1.
I20260928 04:56:59.791512 16150 leader_election.cc:302] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73844ccbbec74d319e8a1548799736c3, f57d763bdbdf4a00b28759301947652e; no voters:
I20260928 04:56:59.791662 16283 raft_consensus.cc:2781] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:56:59.791730 16082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e87e1e16154b529dcfde5cf38eb928" candidate_uuid: "73844ccbbec74d319e8a1548799736c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0930e5479d854889bd0ebe4650232873" is_pre_election: true
I20260928 04:56:59.791890 16283 raft_consensus.cc:686] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [term 1 LEADER]: Becoming Leader. State: Replica: 73844ccbbec74d319e8a1548799736c3, State: Running, Role: LEADER
I20260928 04:56:59.791929 16082 raft_consensus.cc:2454] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73844ccbbec74d319e8a1548799736c3 in term 0.
I20260928 04:56:59.791999 16283 consensus_queue.cc:237] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } } peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } }
I20260928 04:56:59.793036 16082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e87e1e16154b529dcfde5cf38eb928" candidate_uuid: "73844ccbbec74d319e8a1548799736c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0930e5479d854889bd0ebe4650232873"
I20260928 04:56:59.793128 16082 raft_consensus.cc:3037] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:56:59.793187 15803 catalog_manager.cc:5495] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73844ccbbec74d319e8a1548799736c3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "73844ccbbec74d319e8a1548799736c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } health_report { overall_health: UNKNOWN } } }
I20260928 04:56:59.794227 16082 raft_consensus.cc:2454] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73844ccbbec74d319e8a1548799736c3 in term 1.
I20260928 04:56:59.804243 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.801068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59246 (local address 127.0.98.3:42039)
0928 04:56:59.801110 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:56:59.801112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:56:59.801120 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:56:59.801192 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:56:59.801193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:56:59.801211 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:56:59.801277 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:56:59.801453 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.801969 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.803384 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:56:59.803588 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:56:59.803646 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:56:59.803872 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:56:59.804006 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:56:59.804088 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 04:56:59.804145 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 04:56:59.804167 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:56:59.806216 15948 raft_consensus.cc:1261] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Refusing update from remote peer 73844ccbbec74d319e8a1548799736c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:56:59.806224 16082 raft_consensus.cc:1261] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873 [term 1 FOLLOWER]: Refusing update from remote peer 73844ccbbec74d319e8a1548799736c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:56:59.806666 16289 consensus_queue.cc:1035] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:56:59.806846 16283 consensus_queue.cc:1035] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:56:59.809419 16292 mvcc.cc:205] Tried to move back new op lower bound from 7334180535523381248 to 7334180535468556288. Current Snapshot: MvccSnapshot[applied={T|T < 7334180535523381248}]
I20260928 04:56:59.815004 16293 mvcc.cc:205] Tried to move back new op lower bound from 7334180535523381248 to 7334180535468556288. Current Snapshot: MvccSnapshot[applied={T|T < 7334180535523381248}]
I20260928 04:56:59.815735 15802 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:56:59.815846 15802 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:56:59.816757 15802 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: Sending DeleteTablet for 3 replicas of tablet 46e87e1e16154b529dcfde5cf38eb928
I20260928 04:56:59.817214 16196 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e87e1e16154b529dcfde5cf38eb928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:56:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59240
I20260928 04:56:59.817466 16062 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e87e1e16154b529dcfde5cf38eb928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:56:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42390
I20260928 04:56:59.817868 15928 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e87e1e16154b529dcfde5cf38eb928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:56:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40334
I20260928 04:56:59.818185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 15865
I20260928 04:56:59.822686 16301 tablet_replica.cc:330] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3: stopping tablet replica
I20260928 04:56:59.822875 16302 tablet_replica.cc:330] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: stopping tablet replica
I20260928 04:56:59.823078 16302 raft_consensus.cc:2229] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:56:59.823233 16301 raft_consensus.cc:2229] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:56:59.823433 16302 raft_consensus.cc:2258] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:56:59.823599 16301 raft_consensus.cc:2258] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:56:59.824404 16301 ts_tablet_manager.cc:1905] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 04:56:59.825691 15788 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33483 (error 108)
I20260928 04:56:59.826332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 15999
I20260928 04:56:59.826799 16302 ts_tablet_manager.cc:1905] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:56:59.827088 16301 ts_tablet_manager.cc:1918] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:56:59.827222 16301 log.cc:1196] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/46e87e1e16154b529dcfde5cf38eb928
I20260928 04:56:59.827596 16301 ts_tablet_manager.cc:1939] T 46e87e1e16154b529dcfde5cf38eb928 P 73844ccbbec74d319e8a1548799736c3: Deleting consensus metadata
I20260928 04:56:59.828068 15790 catalog_manager.cc:4841] TS 73844ccbbec74d319e8a1548799736c3 (127.0.98.3:42039): tablet 46e87e1e16154b529dcfde5cf38eb928 (table test-table [id=460be993e0924ad78d4ff5ecf4299f0b]) successfully deleted
I20260928 04:56:59.829566 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.829384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33483 (local address 127.0.0.1:40348)
0928 04:56:59.829466 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:59.829523 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 04:56:59.829799 15788 catalog_manager.cc:4584] TS f57d763bdbdf4a00b28759301947652e (127.0.98.1:33483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 46e87e1e16154b529dcfde5cf38eb928: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33483: connect: Connection refused (error 111)
I20260928 04:56:59.831992 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 16133
I20260928 04:56:59.832091 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:56:59.831943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34075 (local address 127.0.0.1:42400)
0928 04:56:59.832013 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 04:56:59.832054 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 04:56:59.837340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 15771
I20260928 04:57:19.843123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43201
--webserver_interface=127.0.98.62
--webserver_port=39071
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:57:19.926549 16305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:57:19.926780 16305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:57:19.926805 16305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:57:19.928184 16305 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 04:57:19.928247 16305 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 04:57:19.928262 16305 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 04:57:19.928275 16305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:57:19.928289 16305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:57:19.928303 16305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:57:19.928314 16305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:57:19.928328 16305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:57:19.929844 16305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:57:19.930059 16305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:57:19.930245 16305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:57:19.939781 16312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:19.939834 16314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:19.939844 16311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:19.939869 16305 server_base.cc:1034] running on GCE node
I20260928 04:57:19.940207 16305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:57:19.940610 16305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:57:19.941769 16305 hybrid_clock.cc:648] HybridClock initialized: now 1790571439941756 us; error 29 us; skew 500 ppm
Sep 28 04:57:19 dist-test-slave-m39l krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:57:19.948807 16305 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:57:19.949361 16305 webserver.cc:460] Webserver started at http://127.0.98.62:39071/ using document root <none> and password file <none>
I20260928 04:57:19.949615 16305 fs_manager.cc:359] Metadata directory not provided
I20260928 04:57:19.949705 16305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:57:19.951292 16305 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:57:19.952096 16321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:19.952409 16305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:57:19.952520 16305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e"
format_stamp: "Formatted at 2026-09-28 04:56:58 on dist-test-slave-m39l"
server_key: "3c9d3abd5d00152068d5979ded4f764d"
server_key_iv: "5a2044b69944d6842959eb24da01ed27"
server_key_version: "encryptionkey@0"
I20260928 04:57:19.952943 16305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:57:19.979274 16305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:57:19.981460 16305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:57:19.981639 16305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:57:19.986006 16305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43201
I20260928 04:57:19.986047 16373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43201 every 8 connection(s)
I20260928 04:57:19.986474 16305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:57:19.988024 16374 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:57:19.988734 16374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: Bootstrap starting.
I20260928 04:57:19.988977 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 16305
I20260928 04:57:19.989266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33483
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43201
--webserver_port=40287
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:57:19.997609 16374 log.cc:824] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:57:19.999852 16374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:57:20.000165 16374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: Bootstrap complete.
I20260928 04:57:20.002803 16374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } }
I20260928 04:57:20.003086 16374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cdfd8e2733a4f2d9afb456cdc3ed10e, State: Initialized, Role: FOLLOWER
I20260928 04:57:20.003280 16374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } }
I20260928 04:57:20.003360 16374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:57:20.003422 16374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:57:20.003494 16374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 1 FOLLOWER]: Advancing to term 2
I20260928 04:57:20.004599 16374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } }
I20260928 04:57:20.004733 16374 leader_election.cc:302] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cdfd8e2733a4f2d9afb456cdc3ed10e; no voters:
I20260928 04:57:20.004971 16374 leader_election.cc:288] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 04:57:20.005007 16378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 04:57:20.005126 16378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 2 LEADER]: Becoming Leader. State: Replica: 9cdfd8e2733a4f2d9afb456cdc3ed10e, State: Running, Role: LEADER
I20260928 04:57:20.005214 16378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } }
I20260928 04:57:20.005259 16374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:57:20.005599 16380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cdfd8e2733a4f2d9afb456cdc3ed10e. Latest consensus state: current_term: 2 leader_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } } }
I20260928 04:57:20.005628 16379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } } }
I20260928 04:57:20.005776 16380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [sys.catalog]: This master's current role is: LEADER
I20260928 04:57:20.005796 16379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [sys.catalog]: This master's current role is: LEADER
I20260928 04:57:20.006469 16390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:57:20.007562 16390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:57:20.007663 16390 catalog_manager.cc:1259] Loaded cluster ID: a8b3f961a235402cbaa7aff44d891532
I20260928 04:57:20.007694 16390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:57:20.008877 16390 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:57:20.009061 16390 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: Loaded TSK: 0
I20260928 04:57:20.019657 16390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: Generated new TSK 1
I20260928 04:57:20.019784 16390 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 04:57:20.081149 16376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:57:20.081422 16376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:57:20.081465 16376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:57:20.082724 16376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:57:20.082809 16376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:57:20.082855 16376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:57:20.084450 16376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40287
--tserver_master_addrs=127.0.98.62:43201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:57:20.084725 16376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:57:20.084973 16376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:57:20.090166 16402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:20.090226 16401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:20.090294 16404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:20.090689 16376 server_base.cc:1034] running on GCE node
I20260928 04:57:20.090950 16376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:57:20.091233 16376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:57:20.092424 16376 hybrid_clock.cc:648] HybridClock initialized: now 1790571440092427 us; error 55 us; skew 500 ppm
Sep 28 04:57:20 dist-test-slave-m39l krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:57:20.095677 16376 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:57:20.096261 16376 webserver.cc:460] Webserver started at http://127.0.98.1:40287/ using document root <none> and password file <none>
I20260928 04:57:20.096544 16376 fs_manager.cc:359] Metadata directory not provided
I20260928 04:57:20.096635 16376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:57:20.098284 16376 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:57:20.099269 16411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:20.099511 16376 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:57:20.099617 16376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "f57d763bdbdf4a00b28759301947652e"
format_stamp: "Formatted at 2026-09-28 04:56:59 on dist-test-slave-m39l"
server_key: "1d43a86e45ffe46192c31ad5c21e127e"
server_key_iv: "b557847cb6f680a77b09c8b95b3d8422"
server_key_version: "encryptionkey@0"
I20260928 04:57:20.100106 16376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:57:20.121130 16376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:57:20.125785 16376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:57:20.126143 16376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:57:20.126459 16376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:57:20.127087 16418 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 04:57:20.129256 16376 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 04:57:20.129401 16376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260928 04:57:20.129448 16376 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 04:57:20.130645 16376 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 04:57:20.130697 16376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260928 04:57:20.130785 16418 tablet_bootstrap.cc:492] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e: Bootstrap starting.
I20260928 04:57:20.139364 16376 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33483
I20260928 04:57:20.139417 16525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33483 every 8 connection(s)
I20260928 04:57:20.139878 16376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:57:20.142791 16418 log.cc:824] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:57:20.144835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 16376
I20260928 04:57:20.145146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34075
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43201
--webserver_port=34125
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:57:20.145897 16418 tablet_bootstrap.cc:492] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:57:20.146188 16418 tablet_bootstrap.cc:492] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e: Bootstrap complete.
I20260928 04:57:20.147432 16418 ts_tablet_manager.cc:1397] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.012s
I20260928 04:57:20.148820 16418 raft_consensus.cc:348] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } } peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } }
I20260928 04:57:20.149184 16418 raft_consensus.cc:729] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f57d763bdbdf4a00b28759301947652e, State: Initialized, Role: FOLLOWER
I20260928 04:57:20.149341 16418 consensus_queue.cc:260] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } } peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } }
I20260928 04:57:20.149657 16418 ts_tablet_manager.cc:1428] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
Sep 28 04:57:20 dist-test-slave-m39l krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:57:20.154913 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.140937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35473 (local address 127.0.98.62:43201)
0928 04:57:20.143135 (+ 2198us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:20.143146 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:20.143194 (+ 48us) server_negotiation.cc:407] Connection header received
0928 04:57:20.143355 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:20.143364 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:20.143442 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:20.143599 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:57:20.144515 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.145945 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.147550 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.148168 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.148282 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.151180 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:57:20.151184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:57:20.151185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:57:20.151198 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:57:20.153002 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:20.153477 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:20.153481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:20.153481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:20.153593 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:20.153877 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:20.153878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:20.153878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:20.154127 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:57:20.154225 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:20.154566 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 04:57:20.154709 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2129,"thread_start_us":111,"threads_started":1}
I20260928 04:57:20.154928 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.140800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:35473)
0928 04:57:20.141009 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:20.141066 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:20.141818 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:20.143659 (+ 1841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:20.143664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:20.144059 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:57:20.144350 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:20.144357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.146078 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.146080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:20.147424 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:20.147428 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.148284 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.148286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:20.148427 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.149016 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:57:20.149027 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:57:20.151055 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:57:20.153114 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:57:20.153123 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:57:20.153125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:57:20.153381 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:57:20.153672 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:57:20.153673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:57:20.153673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:57:20.153791 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:57:20.154203 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:57:20.154210 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:57:20.154365 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 04:57:20.154479 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 04:57:20.154703 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":104,"threads_started":1}
I20260928 04:57:20.156060 16526 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43201
I20260928 04:57:20.156188 16526 heartbeater.cc:464] Registering TS with master...
I20260928 04:57:20.156385 16526 heartbeater.cc:511] Master 127.0.98.62:43201 requested a full tablet report, sending...
I20260928 04:57:20.157151 16338 ts_manager.cc:194] Registered new tserver with Master: f57d763bdbdf4a00b28759301947652e (127.0.98.1:33483)
I20260928 04:57:20.158474 16338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35473
I20260928 04:57:20.159415 16526 heartbeater.cc:503] Master 127.0.98.62:43201 was elected leader, sending a full tablet report...
I20260928 04:57:20.171731 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.157802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33483 (local address 127.0.0.1:52122)
0928 04:57:20.162749 (+ 4947us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:20.162800 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:20.162891 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:20.166986 (+ 4095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:20.166992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:20.166996 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:57:20.167114 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:20.167118 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.168589 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.168591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:20.170109 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:20.170116 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.171361 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.171363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:20.171465 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.171480 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 04:57:20.171561 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 04:57:20.171629 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4905,"thread_start_us":100,"threads_started":1}
I20260928 04:57:20.171780 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.158064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.0.98.1:33483)
0928 04:57:20.166707 (+ 8643us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:20.166713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:20.166730 (+ 17us) server_negotiation.cc:407] Connection header received
0928 04:57:20.166780 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:20.166782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:20.166803 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:20.166920 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:57:20.167227 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.168431 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.170246 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.171167 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.171247 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.171315 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:20.171608 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 04:57:20.171668 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8575,"thread_start_us":73,"threads_started":1}
I20260928 04:57:20.172297 16460 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e87e1e16154b529dcfde5cf38eb928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:56:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52122
I20260928 04:57:20.173175 16535 tablet_replica.cc:330] stopping tablet replica
I20260928 04:57:20.173346 16535 raft_consensus.cc:2229] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:57:20.173429 16535 raft_consensus.cc:2258] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:57:20.173933 16535 ts_tablet_manager.cc:1905] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:57:20.176746 16535 ts_tablet_manager.cc:1918] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:57:20.176832 16535 log.cc:1196] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/wal/wals/46e87e1e16154b529dcfde5cf38eb928
I20260928 04:57:20.177078 16535 ts_tablet_manager.cc:1939] T 46e87e1e16154b529dcfde5cf38eb928 P f57d763bdbdf4a00b28759301947652e: Deleting consensus metadata
I20260928 04:57:20.177732 16323 catalog_manager.cc:4841] TS f57d763bdbdf4a00b28759301947652e (127.0.98.1:33483): tablet 46e87e1e16154b529dcfde5cf38eb928 (table test-table [id=460be993e0924ad78d4ff5ecf4299f0b]) successfully deleted
W20260928 04:57:20.248986 16531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:57:20.249207 16531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:57:20.249233 16531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:57:20.250353 16531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:57:20.250389 16531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:57:20.250417 16531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:57:20.251875 16531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34125
--tserver_master_addrs=127.0.98.62:43201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:57:20.252066 16531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:57:20.252250 16531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:57:20.257561 16541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:20.257565 16543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:20.257561 16540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:20.257587 16531 server_base.cc:1034] running on GCE node
I20260928 04:57:20.258018 16531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:57:20.258234 16531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:57:20.259428 16531 hybrid_clock.cc:648] HybridClock initialized: now 1790571440259419 us; error 32 us; skew 500 ppm
Sep 28 04:57:20 dist-test-slave-m39l krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:57:20.262014 16531 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:57:20.262445 16531 webserver.cc:460] Webserver started at http://127.0.98.2:34125/ using document root <none> and password file <none>
I20260928 04:57:20.262720 16531 fs_manager.cc:359] Metadata directory not provided
I20260928 04:57:20.262781 16531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:57:20.264139 16531 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:57:20.264927 16550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:20.265120 16531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:57:20.265208 16531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "0930e5479d854889bd0ebe4650232873"
format_stamp: "Formatted at 2026-09-28 04:56:59 on dist-test-slave-m39l"
server_key: "28bd1ca9f5172e33acd9ee3df0a9ff5d"
server_key_iv: "61a7147a294539278f8421286702090c"
server_key_version: "encryptionkey@0"
I20260928 04:57:20.265664 16531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:57:20.285567 16531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:57:20.288266 16531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:57:20.288393 16531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:57:20.288604 16531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:57:20.289156 16557 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 04:57:20.290766 16531 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 04:57:20.290827 16531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 04:57:20.290859 16531 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 04:57:20.291565 16531 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 04:57:20.291602 16531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 04:57:20.291810 16557 tablet_bootstrap.cc:492] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: Bootstrap starting.
I20260928 04:57:20.293934 16557 log.cc:824] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: Log is configured to *not* fsync() on all Append() calls
I20260928 04:57:20.297554 16557 tablet_bootstrap.cc:492] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:57:20.297943 16557 tablet_bootstrap.cc:492] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: Bootstrap complete.
I20260928 04:57:20.298131 16557 ts_tablet_manager.cc:1397] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 04:57:20.298892 16531 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34075
I20260928 04:57:20.298987 16665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34075 every 8 connection(s)
I20260928 04:57:20.299260 16531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:57:20.300704 16557 raft_consensus.cc:348] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } } peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } }
I20260928 04:57:20.301052 16557 raft_consensus.cc:729] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0930e5479d854889bd0ebe4650232873, State: Initialized, Role: FOLLOWER
I20260928 04:57:20.301235 16557 consensus_queue.cc:260] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } } peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } }
I20260928 04:57:20.301517 16557 ts_tablet_manager.cc:1428] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: Time spent starting tablet: real 0.003s user 0.007s sys 0.000s
I20260928 04:57:20.304437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 16531
I20260928 04:57:20.304746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42039
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43201
--webserver_port=40209
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:57:20 dist-test-slave-m39l krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:57:20.310322 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.300074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40969 (local address 127.0.98.62:43201)
0928 04:57:20.300131 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:20.300133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:20.300409 (+ 276us) server_negotiation.cc:407] Connection header received
0928 04:57:20.300807 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:20.300808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:20.300827 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:20.300890 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:57:20.301811 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.302328 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.303635 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.303890 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.303957 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.307201 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:57:20.307203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:57:20.307204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:57:20.307212 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:57:20.308734 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:20.309168 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:20.309170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:20.309170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:20.309234 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:20.309564 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:20.309566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:20.309566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:20.309695 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:57:20.309817 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:20.310118 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 04:57:20.310220 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:57:20.310658 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.299977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:40969)
0928 04:57:20.300214 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:20.300261 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:20.300674 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:20.300963 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:20.300967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:20.301419 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:57:20.301703 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:20.301709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.302438 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.302438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:20.303535 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:20.303537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.303976 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.303977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:20.304082 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.304898 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:57:20.304909 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:57:20.307084 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:57:20.308832 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:57:20.308841 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:57:20.308844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:57:20.309085 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:57:20.309320 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:57:20.309321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:57:20.309321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:57:20.309465 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:57:20.309806 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:57:20.309813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:57:20.309973 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 04:57:20.310084 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 04:57:20.310259 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":103,"threads_started":1}
I20260928 04:57:20.311194 16666 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43201
I20260928 04:57:20.311321 16666 heartbeater.cc:464] Registering TS with master...
I20260928 04:57:20.311539 16666 heartbeater.cc:511] Master 127.0.98.62:43201 requested a full tablet report, sending...
I20260928 04:57:20.312088 16338 ts_manager.cc:194] Registered new tserver with Master: 0930e5479d854889bd0ebe4650232873 (127.0.98.2:34075)
I20260928 04:57:20.312662 16338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40969
I20260928 04:57:20.313730 16666 heartbeater.cc:503] Master 127.0.98.62:43201 was elected leader, sending a full tablet report...
Sep 28 04:57:20 dist-test-slave-m39l krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260928 04:57:20.321668 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.312387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34075 (local address 127.0.0.1:53388)
0928 04:57:20.312437 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:20.312448 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:20.312483 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:20.312907 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:20.312908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:20.313214 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:57:20.313251 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:20.313253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.314648 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.314649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:20.315137 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:20.315138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.315827 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.315829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:20.315875 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.316424 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:57:20.316429 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:57:20.318045 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:57:20.320368 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:57:20.320377 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:57:20.320379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:57:20.320609 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:57:20.320910 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:57:20.320911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:57:20.320911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:57:20.320975 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:57:20.321353 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:57:20.321361 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:57:20.321420 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 04:57:20.321491 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 04:57:20.321595 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 04:57:20.321765 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.312482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.0.98.2:34075)
0928 04:57:20.312680 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:20.312685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:20.312697 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:57:20.312783 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:20.312786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:20.312803 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:20.312904 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:57:20.313343 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.314565 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.315222 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.315739 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.315816 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.318155 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:57:20.318165 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:57:20.318166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:57:20.318174 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:57:20.320231 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:20.320722 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:20.320727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:20.320727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:20.320827 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:20.321043 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:20.321044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:20.321045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:20.321260 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:57:20.321350 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:20.321543 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 04:57:20.321651 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":99,"threads_started":1}
I20260928 04:57:20.322291 16600 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e87e1e16154b529dcfde5cf38eb928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:56:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53388
I20260928 04:57:20.323004 16672 tablet_replica.cc:330] stopping tablet replica
I20260928 04:57:20.323138 16672 raft_consensus.cc:2229] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:57:20.323226 16672 raft_consensus.cc:2258] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:57:20.323549 16672 ts_tablet_manager.cc:1905] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:57:20.326138 16672 ts_tablet_manager.cc:1918] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:57:20.326334 16672 log.cc:1196] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/46e87e1e16154b529dcfde5cf38eb928
I20260928 04:57:20.326745 16672 ts_tablet_manager.cc:1939] T 46e87e1e16154b529dcfde5cf38eb928 P 0930e5479d854889bd0ebe4650232873: Deleting consensus metadata
I20260928 04:57:20.327464 16322 catalog_manager.cc:4841] TS 0930e5479d854889bd0ebe4650232873 (127.0.98.2:34075): tablet 46e87e1e16154b529dcfde5cf38eb928 (table test-table [id=460be993e0924ad78d4ff5ecf4299f0b]) successfully deleted
W20260928 04:57:20.393888 16670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:57:20.394124 16670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:57:20.394150 16670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:57:20.395475 16670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:57:20.395535 16670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:57:20.395568 16670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:57:20.397047 16670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40209
--tserver_master_addrs=127.0.98.62:43201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:57:20.397267 16670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:57:20.397455 16670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:57:20.403344 16680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:20.403453 16678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:20.403685 16670 server_base.cc:1034] running on GCE node
W20260928 04:57:20.403702 16677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:20.403991 16670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:57:20.404311 16670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:57:20.405507 16670 hybrid_clock.cc:648] HybridClock initialized: now 1790571440405464 us; error 67 us; skew 500 ppm
Sep 28 04:57:20 dist-test-slave-m39l krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:57:20.408370 16670 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:57:20.408850 16670 webserver.cc:460] Webserver started at http://127.0.98.3:40209/ using document root <none> and password file <none>
I20260928 04:57:20.409049 16670 fs_manager.cc:359] Metadata directory not provided
I20260928 04:57:20.409114 16670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:57:20.410547 16670 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:57:20.411320 16687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:20.411665 16670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:57:20.411759 16670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "73844ccbbec74d319e8a1548799736c3"
format_stamp: "Formatted at 2026-09-28 04:56:59 on dist-test-slave-m39l"
server_key: "aa55783c70742e51ad1572a8c3762208"
server_key_iv: "ed2cae04595b783e44f399ba809d7e2b"
server_key_version: "encryptionkey@0"
I20260928 04:57:20.412158 16670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:57:20.436115 16670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:57:20.438591 16670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:57:20.438714 16670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:57:20.438920 16670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:57:20.439268 16670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:57:20.439316 16670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:20.439340 16670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:57:20.439353 16670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:20.445540 16670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42039
I20260928 04:57:20.445598 16800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42039 every 8 connection(s)
I20260928 04:57:20.446003 16670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:57:20.450970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 16670
Sep 28 04:57:20 dist-test-slave-m39l krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:57:20.457870 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.446884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48203 (local address 127.0.98.62:43201)
0928 04:57:20.446981 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:20.446983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:20.447177 (+ 194us) server_negotiation.cc:407] Connection header received
0928 04:57:20.447674 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:20.447676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:20.447687 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:20.447750 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:57:20.448699 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.449208 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.450270 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.450472 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.450702 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.453820 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:57:20.453822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:57:20.453822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:57:20.453832 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:57:20.455706 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:20.456496 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:20.456497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:20.456497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:20.456569 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:20.456963 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:20.456965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:20.456965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:20.457095 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:57:20.457133 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:20.457601 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 04:57:20.457768 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 04:57:20.458019 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.446875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:48203)
0928 04:57:20.447035 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:20.447078 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:20.447574 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:20.447809 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:20.447812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:20.448387 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:57:20.448593 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:20.448597 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.449309 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.449310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:20.450163 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:20.450165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.450748 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.450749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:20.450937 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.451691 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:57:20.451702 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:57:20.453698 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:57:20.455795 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:57:20.455804 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:57:20.455808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:57:20.456203 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:57:20.456673 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:57:20.456673 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:57:20.456674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:57:20.456849 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:57:20.457199 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:57:20.457207 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:57:20.457449 (+ 242us) client_negotiation.cc:769] Sending connection context
0928 04:57:20.457581 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 04:57:20.457777 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260928 04:57:20.458580 16801 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43201
I20260928 04:57:20.458698 16801 heartbeater.cc:464] Registering TS with master...
I20260928 04:57:20.458911 16801 heartbeater.cc:511] Master 127.0.98.62:43201 requested a full tablet report, sending...
I20260928 04:57:20.459430 16338 ts_manager.cc:194] Registered new tserver with Master: 73844ccbbec74d319e8a1548799736c3 (127.0.98.3:42039)
I20260928 04:57:20.459548 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.451694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34530 (local address 127.0.98.62:43201)
0928 04:57:20.451836 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:20.451839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:20.451883 (+ 44us) server_negotiation.cc:407] Connection header received
0928 04:57:20.451909 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:20.451910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:20.451924 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:20.452001 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:57:20.452713 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.453512 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.454609 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.454835 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.454916 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.456391 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:57:20.456392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:57:20.456393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:57:20.456400 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:57:20.458031 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:20.458654 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:20.458656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:20.458656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:20.458720 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:20.458951 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:20.458952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:20.458953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:20.459070 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:57:20.459225 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:20.459312 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 04:57:20.459423 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260928 04:57:20.459924 16338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48203
I20260928 04:57:20.460328 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:57:20.463467 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.460720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.0.98.62:43201)
0928 04:57:20.460757 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:20.460759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:20.460882 (+ 123us) server_negotiation.cc:407] Connection header received
0928 04:57:20.460961 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:20.460962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:20.461003 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:20.461088 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:57:20.461192 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.461711 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.462485 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.462738 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.462822 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.463001 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:57:20.463059 (+ 58us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 04:57:20.463383 (+ 324us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34534: token expired
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 04:57:20.463532 16805 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34534: token expired
W20260928 04:57:20.463687 16265 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:43201)
I20260928 04:57:20.471581 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.463990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.0.98.62:43201)
0928 04:57:20.464032 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:20.464034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:20.464122 (+ 88us) server_negotiation.cc:407] Connection header received
0928 04:57:20.464153 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:20.464154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:20.464166 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:20.464236 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:57:20.465010 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.465766 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.467097 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.467279 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.467339 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.468998 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:57:20.469000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:57:20.469000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:57:20.469007 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:57:20.470546 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:20.470942 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:20.470942 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:20.470943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:20.470994 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:20.471136 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:20.471137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:20.471137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:20.471240 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:57:20.471316 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:20.471417 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 04:57:20.471483 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:57:20.472983 16337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:57:20.474805 16337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:57:20.477234 16600 tablet_service.cc:1467] Processing CreateTablet for tablet 76f6c467545042ba8a3e0146d985b6ee (DEFAULT_TABLE table=test-table [id=d2c11006e41b4f7fa54642bba1b21f6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:57:20.477186 16460 tablet_service.cc:1467] Processing CreateTablet for tablet 76f6c467545042ba8a3e0146d985b6ee (DEFAULT_TABLE table=test-table [id=d2c11006e41b4f7fa54642bba1b21f6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:57:20.477448 16460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f6c467545042ba8a3e0146d985b6ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:57:20.477449 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f6c467545042ba8a3e0146d985b6ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:57:20.479228 16557 tablet_bootstrap.cc:492] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873: Bootstrap starting.
I20260928 04:57:20.479229 16418 tablet_bootstrap.cc:492] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e: Bootstrap starting.
I20260928 04:57:20.479630 16557 tablet_bootstrap.cc:654] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873: Neither blocks nor log segments found. Creating new log.
I20260928 04:57:20.479734 16418 tablet_bootstrap.cc:654] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e: Neither blocks nor log segments found. Creating new log.
I20260928 04:57:20.480463 16418 tablet_bootstrap.cc:492] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e: No bootstrap required, opened a new log
I20260928 04:57:20.480522 16418 ts_tablet_manager.cc:1397] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 04:57:20.480705 16557 tablet_bootstrap.cc:492] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873: No bootstrap required, opened a new log
I20260928 04:57:20.480674 16418 raft_consensus.cc:348] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } } peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } }
I20260928 04:57:20.480762 16418 raft_consensus.cc:374] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:57:20.480779 16557 ts_tablet_manager.cc:1397] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:57:20.480816 16418 raft_consensus.cc:729] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f57d763bdbdf4a00b28759301947652e, State: Initialized, Role: FOLLOWER
I20260928 04:57:20.480860 16418 consensus_queue.cc:260] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } } peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } }
I20260928 04:57:20.480942 16418 ts_tablet_manager.cc:1428] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:20.480957 16557 raft_consensus.cc:348] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } } peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } }
I20260928 04:57:20.481041 16557 raft_consensus.cc:374] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:57:20.481104 16557 raft_consensus.cc:729] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0930e5479d854889bd0ebe4650232873, State: Initialized, Role: FOLLOWER
I20260928 04:57:20.481158 16557 consensus_queue.cc:260] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } } peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } }
I20260928 04:57:20.481374 16557 ts_tablet_manager.cc:1428] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 04:57:20.482688 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.476904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42039 (local address 127.0.0.1:56148)
0928 04:57:20.477004 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:20.477019 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:20.477060 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:20.478792 (+ 1732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:20.478793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:20.478795 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:57:20.478855 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:20.478856 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.480116 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.480117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:20.481283 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:20.481287 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.482367 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.482369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:20.482432 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.482436 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:57:20.482558 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 04:57:20.482612 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 04:57:20.482700 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.476983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56148 (local address 127.0.98.3:42039)
0928 04:57:20.478636 (+ 1653us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:20.478640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:20.478650 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:57:20.478683 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:20.478684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:20.478698 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:20.478795 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:57:20.478920 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.480003 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.481493 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.482267 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.482365 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.482434 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:20.482566 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 04:57:20.482624 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1614,"thread_start_us":105,"threads_started":1}
I20260928 04:57:20.483505 16735 tablet_service.cc:1467] Processing CreateTablet for tablet 76f6c467545042ba8a3e0146d985b6ee (DEFAULT_TABLE table=test-table [id=d2c11006e41b4f7fa54642bba1b21f6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:57:20.483891 16735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f6c467545042ba8a3e0146d985b6ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:57:20.486997 16810 tablet_bootstrap.cc:492] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3: Bootstrap starting.
I20260928 04:57:20.487814 16810 tablet_bootstrap.cc:654] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3: Neither blocks nor log segments found. Creating new log.
I20260928 04:57:20.488374 16810 log.cc:824] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3: Log is configured to *not* fsync() on all Append() calls
I20260928 04:57:20.489210 16810 tablet_bootstrap.cc:492] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3: No bootstrap required, opened a new log
I20260928 04:57:20.489351 16810 ts_tablet_manager.cc:1397] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:57:20.491129 16810 raft_consensus.cc:348] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } } peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } }
I20260928 04:57:20.491328 16810 raft_consensus.cc:374] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:57:20.491381 16810 raft_consensus.cc:729] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73844ccbbec74d319e8a1548799736c3, State: Initialized, Role: FOLLOWER
I20260928 04:57:20.491534 16810 consensus_queue.cc:260] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } } peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } }
I20260928 04:57:20.491844 16810 ts_tablet_manager.cc:1428] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 04:57:20.491887 16801 heartbeater.cc:503] Master 127.0.98.62:43201 was elected leader, sending a full tablet report...
W20260928 04:57:20.549927 16667 tablet.cc:2369] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:57:20.644572 16527 tablet.cc:2369] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:57:20.650586 16812 raft_consensus.cc:482] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:57:20.650732 16812 raft_consensus.cc:504] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } } peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } }
I20260928 04:57:20.651196 16812 leader_election.cc:288] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0930e5479d854889bd0ebe4650232873 (127.0.98.2:34075), f57d763bdbdf4a00b28759301947652e (127.0.98.1:33483)
I20260928 04:57:20.654654 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.651463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33483 (local address 127.0.98.3:58779)
0928 04:57:20.651681 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:20.651694 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:20.651779 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:20.651998 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:20.651999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:20.652001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:57:20.652082 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:20.652084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.652881 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.652882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:20.653679 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:20.653681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.654327 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.654328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:20.654405 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.654410 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:57:20.654481 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 04:57:20.654558 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":17,"thread_start_us":115,"threads_started":1}
I20260928 04:57:20.654662 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.651559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58779 (local address 127.0.98.1:33483)
0928 04:57:20.651611 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:20.651613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:20.651841 (+ 228us) server_negotiation.cc:407] Connection header received
0928 04:57:20.651872 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:20.651873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:20.651888 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:20.651962 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:57:20.652188 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.652762 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.653805 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.654247 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.654309 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.654320 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:20.654530 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 04:57:20.654580 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:57:20.654832 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.651574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47183 (local address 127.0.98.2:34075)
0928 04:57:20.651618 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:20.651620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:20.651629 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:57:20.651662 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:20.651663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:20.651690 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:20.651797 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:57:20.652031 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.652650 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.653848 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.654429 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.654529 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.654617 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:20.654688 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 04:57:20.654731 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:57:20.654796 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.651463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34075 (local address 127.0.98.3:47183)
0928 04:57:20.651525 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:20.651539 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:20.651585 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:20.651861 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:20.651862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:20.651865 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:57:20.651944 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:20.651946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.652757 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.652759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:20.653702 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:20.653704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.654530 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.654532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:20.654619 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.654622 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:57:20.654684 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 04:57:20.654725 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 04:57:20.655238 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f6c467545042ba8a3e0146d985b6ee" candidate_uuid: "73844ccbbec74d319e8a1548799736c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0930e5479d854889bd0ebe4650232873" is_pre_election: true
I20260928 04:57:20.655256 16465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f6c467545042ba8a3e0146d985b6ee" candidate_uuid: "73844ccbbec74d319e8a1548799736c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f57d763bdbdf4a00b28759301947652e" is_pre_election: true
I20260928 04:57:20.655452 16620 raft_consensus.cc:2454] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73844ccbbec74d319e8a1548799736c3 in term 0.
I20260928 04:57:20.655478 16465 raft_consensus.cc:2454] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73844ccbbec74d319e8a1548799736c3 in term 0.
I20260928 04:57:20.655714 16689 leader_election.cc:302] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73844ccbbec74d319e8a1548799736c3, f57d763bdbdf4a00b28759301947652e; no voters:
I20260928 04:57:20.655879 16812 raft_consensus.cc:2781] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:57:20.655937 16812 raft_consensus.cc:482] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:57:20.655966 16812 raft_consensus.cc:3037] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:57:20.656822 16812 raft_consensus.cc:504] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } } peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } }
I20260928 04:57:20.656975 16812 leader_election.cc:288] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [CANDIDATE]: Term 1 election: Requested vote from peers 0930e5479d854889bd0ebe4650232873 (127.0.98.2:34075), f57d763bdbdf4a00b28759301947652e (127.0.98.1:33483)
I20260928 04:57:20.657130 16465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f6c467545042ba8a3e0146d985b6ee" candidate_uuid: "73844ccbbec74d319e8a1548799736c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f57d763bdbdf4a00b28759301947652e"
I20260928 04:57:20.657166 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f6c467545042ba8a3e0146d985b6ee" candidate_uuid: "73844ccbbec74d319e8a1548799736c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0930e5479d854889bd0ebe4650232873"
I20260928 04:57:20.657232 16465 raft_consensus.cc:3037] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:57:20.657332 16620 raft_consensus.cc:3037] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:57:20.658027 16465 raft_consensus.cc:2454] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73844ccbbec74d319e8a1548799736c3 in term 1.
I20260928 04:57:20.658056 16620 raft_consensus.cc:2454] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73844ccbbec74d319e8a1548799736c3 in term 1.
I20260928 04:57:20.658198 16689 leader_election.cc:302] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73844ccbbec74d319e8a1548799736c3, f57d763bdbdf4a00b28759301947652e; no voters:
I20260928 04:57:20.658313 16812 raft_consensus.cc:2781] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:57:20.658527 16812 raft_consensus.cc:686] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [term 1 LEADER]: Becoming Leader. State: Replica: 73844ccbbec74d319e8a1548799736c3, State: Running, Role: LEADER
I20260928 04:57:20.658648 16812 consensus_queue.cc:237] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } } peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } }
I20260928 04:57:20.659559 16338 catalog_manager.cc:5495] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73844ccbbec74d319e8a1548799736c3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "73844ccbbec74d319e8a1548799736c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } health_report { overall_health: UNKNOWN } } }
I20260928 04:57:20.665696 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.663314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56158 (local address 127.0.98.3:42039)
0928 04:57:20.663370 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:20.663372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:20.663380 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:57:20.663436 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:20.663437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:20.663455 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:20.663518 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:57:20.663614 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.664115 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.664838 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:20.665026 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:20.665088 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:20.665246 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:57:20.665404 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:57:20.665482 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:20.665536 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 04:57:20.665555 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 04:57:20.667626 16620 raft_consensus.cc:1261] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873 [term 1 FOLLOWER]: Refusing update from remote peer 73844ccbbec74d319e8a1548799736c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:57:20.667733 16465 raft_consensus.cc:1261] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Refusing update from remote peer 73844ccbbec74d319e8a1548799736c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:57:20.668018 16812 consensus_queue.cc:1035] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:57:20.668148 16814 consensus_queue.cc:1035] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:57:20.670769 16818 mvcc.cc:205] Tried to move back new op lower bound from 7334180620971704320 to 7334180620938448896. Current Snapshot: MvccSnapshot[applied={T|T < 7334180620971704320}]
I20260928 04:57:20.673358 16337 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:57:20.673445 16337 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:57:20.674173 16337 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: Sending DeleteTablet for 3 replicas of tablet 76f6c467545042ba8a3e0146d985b6ee
I20260928 04:57:20.674467 16735 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f6c467545042ba8a3e0146d985b6ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:57:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56148
I20260928 04:57:20.674562 16600 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f6c467545042ba8a3e0146d985b6ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:57:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53388
I20260928 04:57:20.674665 16672 tablet_replica.cc:330] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873: stopping tablet replica
I20260928 04:57:20.674729 16672 raft_consensus.cc:2229] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:57:20.674722 16460 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f6c467545042ba8a3e0146d985b6ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:57:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52122
I20260928 04:57:20.674777 16672 raft_consensus.cc:2258] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:57:20.674796 16826 tablet_replica.cc:330] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3: stopping tablet replica
I20260928 04:57:20.674852 16535 tablet_replica.cc:330] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e: stopping tablet replica
I20260928 04:57:20.674865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 16376
I20260928 04:57:20.674887 16826 raft_consensus.cc:2229] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:57:20.674916 16535 raft_consensus.cc:2229] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:57:20.674968 16535 raft_consensus.cc:2258] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:57:20.675487 16826 raft_consensus.cc:2258] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:57:20.675616 16672 ts_tablet_manager.cc:1905] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:57:20.676800 16826 ts_tablet_manager.cc:1905] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:57:20.677454 16672 ts_tablet_manager.cc:1918] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:57:20.677515 16672 log.cc:1196] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/76f6c467545042ba8a3e0146d985b6ee
I20260928 04:57:20.677732 16672 ts_tablet_manager.cc:1939] T 76f6c467545042ba8a3e0146d985b6ee P 0930e5479d854889bd0ebe4650232873: Deleting consensus metadata
I20260928 04:57:20.678114 16322 catalog_manager.cc:4841] TS 0930e5479d854889bd0ebe4650232873 (127.0.98.2:34075): tablet 76f6c467545042ba8a3e0146d985b6ee (table test-table [id=d2c11006e41b4f7fa54642bba1b21f6c]) successfully deleted
I20260928 04:57:20.678952 16826 ts_tablet_manager.cc:1918] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:57:20.679035 16826 log.cc:1196] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/76f6c467545042ba8a3e0146d985b6ee
I20260928 04:57:20.679325 16826 ts_tablet_manager.cc:1939] T 76f6c467545042ba8a3e0146d985b6ee P 73844ccbbec74d319e8a1548799736c3: Deleting consensus metadata
I20260928 04:57:20.679772 16325 catalog_manager.cc:4841] TS 73844ccbbec74d319e8a1548799736c3 (127.0.98.3:42039): tablet 76f6c467545042ba8a3e0146d985b6ee (table test-table [id=d2c11006e41b4f7fa54642bba1b21f6c]) successfully deleted
W20260928 04:57:20.681808 16323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33483 (error 108)
I20260928 04:57:20.682121 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 16531
I20260928 04:57:20.682364 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:20.682243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33483 (local address 127.0.0.1:52124)
0928 04:57:20.682286 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:20.682325 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 04:57:20.684199 16323 catalog_manager.cc:4584] TS f57d763bdbdf4a00b28759301947652e (127.0.98.1:33483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 76f6c467545042ba8a3e0146d985b6ee: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33483: connect: Connection refused (error 111)
I20260928 04:57:20.687758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 16670
I20260928 04:57:20.692932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 16305
I20260928 04:57:40.698482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43201
--webserver_interface=127.0.98.62
--webserver_port=39071
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:57:40.781299 16828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:57:40.781566 16828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:57:40.781607 16828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:57:40.782888 16828 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 04:57:40.782966 16828 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 04:57:40.782989 16828 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 04:57:40.783007 16828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:57:40.783041 16828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:57:40.783066 16828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:57:40.783095 16828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:57:40.783125 16828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:57:40.784677 16828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:57:40.784971 16828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:57:40.785205 16828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:57:40.790474 16834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:40.790482 16836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:40.790478 16833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:40.790840 16828 server_base.cc:1034] running on GCE node
I20260928 04:57:40.791157 16828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:57:40.791481 16828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:57:40.792680 16828 hybrid_clock.cc:648] HybridClock initialized: now 1790571460792651 us; error 38 us; skew 500 ppm
Sep 28 04:57:40 dist-test-slave-m39l krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:57:40.795667 16828 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:57:40.796178 16828 webserver.cc:460] Webserver started at http://127.0.98.62:39071/ using document root <none> and password file <none>
I20260928 04:57:40.796384 16828 fs_manager.cc:359] Metadata directory not provided
I20260928 04:57:40.796429 16828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:57:40.797789 16828 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:57:40.798482 16843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:40.798705 16828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:57:40.798784 16828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e"
format_stamp: "Formatted at 2026-09-28 04:56:58 on dist-test-slave-m39l"
server_key: "3c9d3abd5d00152068d5979ded4f764d"
server_key_iv: "5a2044b69944d6842959eb24da01ed27"
server_key_version: "encryptionkey@0"
I20260928 04:57:40.799129 16828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:57:40.812968 16828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:57:40.815274 16828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:57:40.815389 16828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:57:40.819346 16828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43201
I20260928 04:57:40.819391 16895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43201 every 8 connection(s)
I20260928 04:57:40.819798 16828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:57:40.821694 16896 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:57:40.822572 16896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: Bootstrap starting.
I20260928 04:57:40.824486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 16828
I20260928 04:57:40.824826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33483
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43201
--webserver_port=40287
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:57:40.833396 16896 log.cc:824] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:57:40.837121 16896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:57:40.837525 16896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: Bootstrap complete.
I20260928 04:57:40.840845 16896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } }
I20260928 04:57:40.841194 16896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cdfd8e2733a4f2d9afb456cdc3ed10e, State: Initialized, Role: FOLLOWER
I20260928 04:57:40.841408 16896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } }
I20260928 04:57:40.841485 16896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:57:40.841523 16896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:57:40.841562 16896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 2 FOLLOWER]: Advancing to term 3
I20260928 04:57:40.842984 16896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } }
I20260928 04:57:40.843102 16896 leader_election.cc:302] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cdfd8e2733a4f2d9afb456cdc3ed10e; no voters:
I20260928 04:57:40.843281 16896 leader_election.cc:288] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 04:57:40.843425 16900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 3 FOLLOWER]: Leader election won for term 3
I20260928 04:57:40.843535 16896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:57:40.843652 16900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [term 3 LEADER]: Becoming Leader. State: Replica: 9cdfd8e2733a4f2d9afb456cdc3ed10e, State: Running, Role: LEADER
I20260928 04:57:40.843791 16900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } }
I20260928 04:57:40.844197 16901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } } }
I20260928 04:57:40.844303 16901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [sys.catalog]: This master's current role is: LEADER
I20260928 04:57:40.844528 16909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:57:40.844872 16902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cdfd8e2733a4f2d9afb456cdc3ed10e. Latest consensus state: current_term: 3 leader_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdfd8e2733a4f2d9afb456cdc3ed10e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43201 } } }
I20260928 04:57:40.844954 16902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e [sys.catalog]: This master's current role is: LEADER
I20260928 04:57:40.845706 16909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:57:40.845853 16909 catalog_manager.cc:1259] Loaded cluster ID: a8b3f961a235402cbaa7aff44d891532
I20260928 04:57:40.846140 16909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:57:40.847443 16909 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:57:40.847612 16909 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: Loaded TSK: 0
I20260928 04:57:40.847667 16909 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9cdfd8e2733a4f2d9afb456cdc3ed10e: Loaded TSK: 1
F20260928 04:57:40.847890 16909 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa170e509ad google::LogMessage::Fail() at ??:0
@ 0x7fa170e5388a google::LogMessage::SendToLog() at ??:0
@ 0x7fa170e504de google::LogMessage::Flush() at ??:0
@ 0x7fa170e51a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa1735bd104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa1735bad41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa178daa1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa178daa48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa178daa594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fa178d70c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fa178d7aad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa172d1e0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa172d13954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa173b776db start_thread at ??:0
@ 0x7fa16f94e61f clone at ??:0
W20260928 04:57:40.934062 16898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:57:40.934312 16898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:57:40.934353 16898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:57:40.935632 16898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:57:40.935706 16898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:57:40.935753 16898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:57:40.937289 16898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40287
--tserver_master_addrs=127.0.98.62:43201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:57:40.937587 16898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:57:40.937844 16898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:57:40.943262 16938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:40.943261 16940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:40.943295 16937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:40.943527 16898 server_base.cc:1034] running on GCE node
I20260928 04:57:40.943789 16898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:57:40.944177 16898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:57:40.945372 16898 hybrid_clock.cc:648] HybridClock initialized: now 1790571460945350 us; error 38 us; skew 500 ppm
Sep 28 04:57:40 dist-test-slave-m39l krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:57:40.948338 16898 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:57:40.948823 16898 webserver.cc:460] Webserver started at http://127.0.98.1:40287/ using document root <none> and password file <none>
I20260928 04:57:40.949126 16898 fs_manager.cc:359] Metadata directory not provided
I20260928 04:57:40.949222 16898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:57:40.950780 16898 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:57:40.951651 16947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:40.951872 16898 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:57:40.951992 16898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "f57d763bdbdf4a00b28759301947652e"
format_stamp: "Formatted at 2026-09-28 04:56:59 on dist-test-slave-m39l"
server_key: "1d43a86e45ffe46192c31ad5c21e127e"
server_key_iv: "b557847cb6f680a77b09c8b95b3d8422"
server_key_version: "encryptionkey@0"
I20260928 04:57:40.952404 16898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:57:40.961410 16898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:57:40.963887 16898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:57:40.964061 16898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:57:40.964310 16898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:57:40.964798 16954 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 04:57:40.966943 16898 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 04:57:40.967018 16898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 04:57:40.967067 16898 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 04:57:40.967818 16898 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 04:57:40.967887 16898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 04:57:40.967955 16954 tablet_bootstrap.cc:492] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e: Bootstrap starting.
I20260928 04:57:40.975317 16898 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33483
I20260928 04:57:40.975785 16898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:57:40.978760 17061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33483 every 8 connection(s)
I20260928 04:57:40.980101 16954 log.cc:824] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:57:40.981405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 16898
I20260928 04:57:40.981701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34075
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43201
--webserver_port=34125
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:57:40.983172 16954 tablet_bootstrap.cc:492] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:57:40.983525 16954 tablet_bootstrap.cc:492] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e: Bootstrap complete.
I20260928 04:57:40.984745 16954 ts_tablet_manager.cc:1397] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20260928 04:57:40.986404 16954 raft_consensus.cc:348] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } } peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } }
I20260928 04:57:40.986862 16954 raft_consensus.cc:729] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f57d763bdbdf4a00b28759301947652e, State: Initialized, Role: FOLLOWER
I20260928 04:57:40.987079 16954 consensus_queue.cc:260] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } } peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } }
I20260928 04:57:40.987421 16954 ts_tablet_manager.cc:1428] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
W20260928 04:57:41.064257 17064 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:57:40.979498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:58371)
0928 04:57:40.979741 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:40.979786 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:40.980458 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:41.064044 (+ 83586us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":114,"threads_started":1}
W20260928 04:57:41.064589 17062 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43201 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43201: Client connection negotiation failed: client connection to 127.0.98.62:43201: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 04:57:41.065001 17064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:41.064822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:42093)
0928 04:57:41.064927 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:41.064975 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 04:57:41.065441 17064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:41.065353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:50813)
0928 04:57:41.065400 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:41.065424 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 04:57:41.065569 17062 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:57:41.069213 17066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:57:41.069438 17066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:57:41.069465 17066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:57:41.070835 17066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:57:41.070890 17066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:57:41.070921 17066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:57:41.072564 17066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34125
--tserver_master_addrs=127.0.98.62:43201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:57:41.072826 17066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:57:41.073096 17066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:57:41.078604 17075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:41.078784 17066 server_base.cc:1034] running on GCE node
W20260928 04:57:41.078604 17072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:41.078647 17073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:41.079107 17066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:57:41.079381 17066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:57:41.080554 17066 hybrid_clock.cc:648] HybridClock initialized: now 1790571461080534 us; error 34 us; skew 500 ppm
Sep 28 04:57:41 dist-test-slave-m39l krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:57:41.083274 17066 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:57:41.083774 17066 webserver.cc:460] Webserver started at http://127.0.98.2:34125/ using document root <none> and password file <none>
I20260928 04:57:41.084017 17066 fs_manager.cc:359] Metadata directory not provided
I20260928 04:57:41.084097 17066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:57:41.085661 17066 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:57:41.086655 17082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:41.086882 17066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:57:41.086979 17066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "0930e5479d854889bd0ebe4650232873"
format_stamp: "Formatted at 2026-09-28 04:56:59 on dist-test-slave-m39l"
server_key: "28bd1ca9f5172e33acd9ee3df0a9ff5d"
server_key_iv: "61a7147a294539278f8421286702090c"
server_key_version: "encryptionkey@0"
I20260928 04:57:41.087380 17066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:57:41.115159 17066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:57:41.117924 17066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:57:41.118269 17066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:57:41.118610 17066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:57:41.118979 17066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:57:41.119037 17066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:41.119073 17066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:57:41.119100 17066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:41.125080 17066 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34075
I20260928 04:57:41.125237 17195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34075 every 8 connection(s)
I20260928 04:57:41.125545 17066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:57:41.127107 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:41.126417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:39419)
0928 04:57:41.126655 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:41.126744 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":142,"threads_started":1}
W20260928 04:57:41.127411 17196 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43201 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43201: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
I20260928 04:57:41.127503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 17066
I20260928 04:57:41.127789 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:41.127681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:39295)
0928 04:57:41.127736 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:41.127763 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 04:57:41.127812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42039
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43201
--webserver_port=40209
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:57:41.128232 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:41.128086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:38505)
0928 04:57:41.128146 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:41.128191 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 04:57:41.128432 17196 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:57:41.211139 17199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:57:41.211382 17199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:57:41.211408 17199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:57:41.212754 17199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:57:41.212805 17199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:57:41.212836 17199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:57:41.214423 17199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40209
--tserver_master_addrs=127.0.98.62:43201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:57:41.214743 17199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:57:41.215003 17199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:57:41.220521 17205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:41.220544 17207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:41.220734 17199 server_base.cc:1034] running on GCE node
W20260928 04:57:41.220549 17204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:41.221122 17199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:57:41.221365 17199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:57:41.222592 17199 hybrid_clock.cc:648] HybridClock initialized: now 1790571461222573 us; error 35 us; skew 500 ppm
Sep 28 04:57:41 dist-test-slave-m39l krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:57:41.225425 17199 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:57:41.225926 17199 webserver.cc:460] Webserver started at http://127.0.98.3:40209/ using document root <none> and password file <none>
I20260928 04:57:41.226176 17199 fs_manager.cc:359] Metadata directory not provided
I20260928 04:57:41.226230 17199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:57:41.227960 17199 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:57:41.228672 17214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:41.228941 17199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:57:41.229034 17199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "73844ccbbec74d319e8a1548799736c3"
format_stamp: "Formatted at 2026-09-28 04:56:59 on dist-test-slave-m39l"
server_key: "aa55783c70742e51ad1572a8c3762208"
server_key_iv: "ed2cae04595b783e44f399ba809d7e2b"
server_key_version: "encryptionkey@0"
I20260928 04:57:41.229466 17199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:57:41.242197 17199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:57:41.244454 17199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:57:41.244583 17199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:57:41.244908 17199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:57:41.245262 17199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:57:41.245306 17199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:41.245332 17199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:57:41.245376 17199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:41.252177 17199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42039
I20260928 04:57:41.252226 17327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42039 every 8 connection(s)
I20260928 04:57:41.252599 17199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:57:41.253491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 17199
I20260928 04:57:41.253813 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:41.253341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:53491)
0928 04:57:41.253528 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:41.253631 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":112,"threads_started":1}
W20260928 04:57:41.254247 17328 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43201 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43201: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
I20260928 04:57:41.254644 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:41.254452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:56579)
0928 04:57:41.254564 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:41.254602 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 04:57:41.255049 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:41.254910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:39503)
0928 04:57:41.254997 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:41.255023 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260928 04:57:41.255168 17328 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 04:57:42.066208 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:42.065968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:33149)
0928 04:57:42.066116 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:42.066158 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260928 04:57:42.129081 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:42.128822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:51777)
0928 04:57:42.128959 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:42.129019 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260928 04:57:42.255934 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:42.255656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:44521)
0928 04:57:42.255819 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:42.255867 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
I20260928 04:57:42.565995 17335 raft_consensus.cc:482] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:57:42.566185 17335 raft_consensus.cc:504] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } } peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } }
I20260928 04:57:42.566917 17335 leader_election.cc:288] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73844ccbbec74d319e8a1548799736c3 (127.0.98.3:42039), 0930e5479d854889bd0ebe4650232873 (127.0.98.2:34075)
Sep 28 04:57:42 dist-test-slave-m39l krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 28 04:57:42 dist-test-slave-m39l krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260928 04:57:42.579003 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:42.566920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39287 (local address 127.0.98.2:34075)
0928 04:57:42.567117 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:42.567129 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:42.567205 (+ 76us) server_negotiation.cc:407] Connection header received
0928 04:57:42.567723 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:42.567726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:42.567771 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:42.567935 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:57:42.568786 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:42.569741 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:42.570885 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:42.571325 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:42.571458 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:42.574327 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:57:42.574331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:57:42.574332 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:57:42.574344 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:57:42.576750 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:42.577513 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:42.577517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:42.577518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:42.577648 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:42.578024 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:42.578025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:42.578026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:42.578391 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:57:42.578507 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:42.578750 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 04:57:42.578875 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260928 04:57:42.578989 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:42.566878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42039 (local address 127.0.98.1:58141)
0928 04:57:42.567164 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:42.567175 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:42.567232 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:42.567972 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:42.567974 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:42.568425 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:57:42.568634 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:42.568640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:42.569776 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:42.569777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:42.570757 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:42.570759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:42.571454 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:42.571455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:42.571578 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:42.572310 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:57:42.572314 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:57:42.574404 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:57:42.576932 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:57:42.576933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:57:42.576933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:57:42.577400 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:57:42.577752 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:57:42.577753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:57:42.577753 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:57:42.577883 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:57:42.578349 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:57:42.578357 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:57:42.578536 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 04:57:42.578669 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 04:57:42.578888 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":23,"thread_start_us":85,"threads_started":1}
I20260928 04:57:42.579226 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:42.566812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34075 (local address 127.0.98.1:39287)
0928 04:57:42.567074 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:42.567086 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:42.567150 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:42.567970 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:42.567973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:42.568423 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:57:42.568635 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:42.568641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:42.569856 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:42.569856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:42.570763 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:42.570766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:42.571445 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:42.571446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:42.571572 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:42.572296 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:57:42.572306 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:57:42.574186 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:57:42.576872 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:57:42.576917 (+ 45us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:57:42.576925 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:57:42.577400 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:57:42.577764 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:57:42.577765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:57:42.577765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:57:42.577902 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:57:42.578474 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:57:42.578475 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:57:42.578567 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 04:57:42.578669 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 04:57:42.578976 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":335,"thread_start_us":69,"threads_started":1}
I20260928 04:57:42.579015 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:42.567055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58141 (local address 127.0.98.3:42039)
0928 04:57:42.567244 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:42.567252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:42.567316 (+ 64us) server_negotiation.cc:407] Connection header received
0928 04:57:42.567730 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:42.567732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:42.567771 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:42.567926 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:57:42.568784 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:42.569657 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:42.570888 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:42.571337 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:42.571479 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:42.574562 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:57:42.574567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:57:42.574567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:57:42.574579 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:57:42.576839 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:42.577514 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:42.577517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:42.577518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:42.577669 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:42.577955 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:42.577956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:42.577957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:42.578258 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:57:42.578341 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:42.578732 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 04:57:42.578873 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":80,"threads_started":1}
W20260928 04:57:42.580118 16951 leader_election.cc:341] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73844ccbbec74d319e8a1548799736c3 (127.0.98.3:42039): Not found: Tablet not found: 76f6c467545042ba8a3e0146d985b6ee
W20260928 04:57:42.580348 16948 leader_election.cc:341] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0930e5479d854889bd0ebe4650232873 (127.0.98.2:34075): Not found: Tablet not found: 76f6c467545042ba8a3e0146d985b6ee
I20260928 04:57:42.580415 16948 leader_election.cc:302] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f57d763bdbdf4a00b28759301947652e; no voters: 0930e5479d854889bd0ebe4650232873, 73844ccbbec74d319e8a1548799736c3
I20260928 04:57:42.580515 17335 raft_consensus.cc:2726] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:57:43.067060 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:43.066855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:40317)
0928 04:57:43.066961 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:43.067015 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 04:57:43.130025 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:43.129792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:46307)
0928 04:57:43.129934 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:43.129980 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260928 04:57:43.256796 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:43.256535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:57443)
0928 04:57:43.256697 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:43.256747 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":64,"threads_started":1}
I20260928 04:57:44.068070 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:44.067847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:58391)
0928 04:57:44.067978 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:44.068023 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260928 04:57:44.130965 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:44.130686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:40439)
0928 04:57:44.130863 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:44.130914 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":69,"threads_started":1}
I20260928 04:57:44.257612 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:44.257348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:40063)
0928 04:57:44.257513 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:44.257560 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20260928 04:57:44.449659 17345 raft_consensus.cc:482] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:57:44.449846 17345 raft_consensus.cc:504] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } } peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } }
I20260928 04:57:44.450088 17345 leader_election.cc:288] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73844ccbbec74d319e8a1548799736c3 (127.0.98.3:42039), 0930e5479d854889bd0ebe4650232873 (127.0.98.2:34075)
W20260928 04:57:44.450600 16951 leader_election.cc:341] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73844ccbbec74d319e8a1548799736c3 (127.0.98.3:42039): Not found: Tablet not found: 76f6c467545042ba8a3e0146d985b6ee
W20260928 04:57:44.450691 16948 leader_election.cc:341] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0930e5479d854889bd0ebe4650232873 (127.0.98.2:34075): Not found: Tablet not found: 76f6c467545042ba8a3e0146d985b6ee
I20260928 04:57:44.450829 16948 leader_election.cc:302] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f57d763bdbdf4a00b28759301947652e; no voters: 0930e5479d854889bd0ebe4650232873, 73844ccbbec74d319e8a1548799736c3
I20260928 04:57:44.450942 17345 raft_consensus.cc:2726] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:57:45.068843 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:45.068608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:51071)
0928 04:57:45.068747 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:45.068794 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260928 04:57:45.131834 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:45.131566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:48927)
0928 04:57:45.131739 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:45.131787 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":76,"threads_started":1}
I20260928 04:57:45.258705 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:45.258344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:59467)
0928 04:57:45.258604 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:45.258658 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":73,"threads_started":1}
I20260928 04:57:46.069729 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:46.069476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:39913)
0928 04:57:46.069632 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:46.069681 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
I20260928 04:57:46.132833 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:46.132510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:59425)
0928 04:57:46.132682 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:46.132736 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":62,"threads_started":1}
I20260928 04:57:46.259577 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:46.259305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:55415)
0928 04:57:46.259484 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:46.259530 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
I20260928 04:57:46.655345 17352 raft_consensus.cc:482] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:57:46.655448 17352 raft_consensus.cc:504] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } } peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } }
I20260928 04:57:46.655638 17352 leader_election.cc:288] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73844ccbbec74d319e8a1548799736c3 (127.0.98.3:42039), 0930e5479d854889bd0ebe4650232873 (127.0.98.2:34075)
W20260928 04:57:46.656077 16951 leader_election.cc:341] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73844ccbbec74d319e8a1548799736c3 (127.0.98.3:42039): Not found: Tablet not found: 76f6c467545042ba8a3e0146d985b6ee
W20260928 04:57:46.656215 16948 leader_election.cc:341] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0930e5479d854889bd0ebe4650232873 (127.0.98.2:34075): Not found: Tablet not found: 76f6c467545042ba8a3e0146d985b6ee
I20260928 04:57:46.656306 16948 leader_election.cc:302] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f57d763bdbdf4a00b28759301947652e; no voters: 0930e5479d854889bd0ebe4650232873, 73844ccbbec74d319e8a1548799736c3
I20260928 04:57:46.656380 17352 raft_consensus.cc:2726] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:57:47.070626 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:47.070347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:55897)
0928 04:57:47.070530 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:47.070578 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
I20260928 04:57:47.133771 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:47.133510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:53885)
0928 04:57:47.133676 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:47.133723 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":63,"threads_started":1}
I20260928 04:57:47.260543 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:47.260288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:60861)
0928 04:57:47.260441 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:47.260488 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260928 04:57:48.071491 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:48.071238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:60765)
0928 04:57:48.071401 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:48.071445 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
I20260928 04:57:48.134712 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:48.134401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:38445)
0928 04:57:48.134604 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:48.134645 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":105,"threads_started":1}
I20260928 04:57:48.261413 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:48.261174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:51577)
0928 04:57:48.261315 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:48.261364 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20260928 04:57:49.072412 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:49.072088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:42339)
0928 04:57:49.072313 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:49.072362 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":117,"threads_started":1}
I20260928 04:57:49.135663 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:49.135384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:38707)
0928 04:57:49.135559 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:49.135615 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
I20260928 04:57:49.262347 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:49.262063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:41921)
0928 04:57:49.262246 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:49.262298 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":65,"threads_started":1}
I20260928 04:57:50.073328 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:50.073080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:43377)
0928 04:57:50.073227 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:50.073278 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260928 04:57:50.136489 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:50.136215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:47755)
0928 04:57:50.136389 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:50.136439 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
I20260928 04:57:50.263226 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:50.262967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:36935)
0928 04:57:50.263125 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:50.263175 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260928 04:57:50.668933 17365 raft_consensus.cc:482] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:57:50.669039 17365 raft_consensus.cc:504] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } } peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } }
I20260928 04:57:50.669224 17365 leader_election.cc:288] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73844ccbbec74d319e8a1548799736c3 (127.0.98.3:42039), 0930e5479d854889bd0ebe4650232873 (127.0.98.2:34075)
W20260928 04:57:50.669690 16948 leader_election.cc:341] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0930e5479d854889bd0ebe4650232873 (127.0.98.2:34075): Not found: Tablet not found: 76f6c467545042ba8a3e0146d985b6ee
W20260928 04:57:50.669751 16951 leader_election.cc:341] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73844ccbbec74d319e8a1548799736c3 (127.0.98.3:42039): Not found: Tablet not found: 76f6c467545042ba8a3e0146d985b6ee
I20260928 04:57:50.669781 16948 leader_election.cc:302] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f57d763bdbdf4a00b28759301947652e; no voters: 0930e5479d854889bd0ebe4650232873, 73844ccbbec74d319e8a1548799736c3
I20260928 04:57:50.669849 17365 raft_consensus.cc:2726] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:57:51.074221 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:51.073938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:56869)
0928 04:57:51.074124 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:51.074172 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
I20260928 04:57:51.137356 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:51.137095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:53417)
0928 04:57:51.137261 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:51.137308 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
I20260928 04:57:51.264108 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:51.263862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:35215)
0928 04:57:51.264010 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:51.264059 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260928 04:57:52.075119 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:52.074847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:42871)
0928 04:57:52.075007 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:52.075067 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260928 04:57:52.138293 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:52.137963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:37449)
0928 04:57:52.138194 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:52.138244 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":77,"threads_started":1}
I20260928 04:57:52.264977 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:52.264700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:47871)
0928 04:57:52.264880 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:52.264928 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":88,"threads_started":1}
I20260928 04:57:53.075907 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:53.075672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:34025)
0928 04:57:53.075816 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:53.075861 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260928 04:57:53.139153 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:53.138901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:46053)
0928 04:57:53.139050 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:53.139102 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260928 04:57:53.160985 17374 raft_consensus.cc:482] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:57:53.161119 17374 raft_consensus.cc:504] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73844ccbbec74d319e8a1548799736c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42039 } } peers { permanent_uuid: "0930e5479d854889bd0ebe4650232873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34075 } } peers { permanent_uuid: "f57d763bdbdf4a00b28759301947652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33483 } }
I20260928 04:57:53.161334 17374 leader_election.cc:288] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73844ccbbec74d319e8a1548799736c3 (127.0.98.3:42039), 0930e5479d854889bd0ebe4650232873 (127.0.98.2:34075)
W20260928 04:57:53.161772 16951 leader_election.cc:341] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73844ccbbec74d319e8a1548799736c3 (127.0.98.3:42039): Not found: Tablet not found: 76f6c467545042ba8a3e0146d985b6ee
W20260928 04:57:53.161825 16948 leader_election.cc:341] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0930e5479d854889bd0ebe4650232873 (127.0.98.2:34075): Not found: Tablet not found: 76f6c467545042ba8a3e0146d985b6ee
I20260928 04:57:53.161852 16951 leader_election.cc:302] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f57d763bdbdf4a00b28759301947652e; no voters: 0930e5479d854889bd0ebe4650232873, 73844ccbbec74d319e8a1548799736c3
I20260928 04:57:53.161917 17374 raft_consensus.cc:2726] T 76f6c467545042ba8a3e0146d985b6ee P f57d763bdbdf4a00b28759301947652e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:57:53.265916 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:53.265645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:37853)
0928 04:57:53.265817 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:53.265865 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":101,"threads_started":1}
I20260928 04:57:54.076709 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:54.076450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:47979)
0928 04:57:54.076613 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:54.076656 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":68,"threads_started":1}
I20260928 04:57:54.139988 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:54.139738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:39485)
0928 04:57:54.139895 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:54.139940 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260928 04:57:54.266741 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:54.266432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:43329)
0928 04:57:54.266608 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:54.266657 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":105,"threads_started":1}
I20260928 04:57:55.077571 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:55.077353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:58161)
0928 04:57:55.077481 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:55.077522 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260928 04:57:55.140836 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:55.140608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:49225)
0928 04:57:55.140746 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:55.140787 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260928 04:57:55.267719 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:55.267356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:56557)
0928 04:57:55.267606 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:55.267658 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":110,"threads_started":1}
I20260928 04:57:56.078621 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:56.078333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.1:37717)
0928 04:57:56.078468 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:56.078537 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260928 04:57:56.141790 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:56.141531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.2:40643)
0928 04:57:56.141700 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:56.141744 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
I20260928 04:57:56.259616 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 16898
I20260928 04:57:56.264742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 17066
I20260928 04:57:56.268558 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:56.268335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43201 (local address 127.0.98.3:38957)
0928 04:57:56.268471 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:56.268513 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260928 04:57:56.269404 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 17199
2026-09-28T04:57:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58761 ms)
[----------] 30 tests from SecurityITest (267863 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:57:56 dist-test-slave-m39l krb5kdc[17387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:57:56 dist-test-slave-m39l krb5kdc[17387](info): set up 2 sockets
Sep 28 04:57:56 dist-test-slave-m39l krb5kdc[17387](info): commencing operation
krb5kdc: starting...
W20260928 04:57:57.318387 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:57:57 dist-test-slave-m39l krb5kdc[17387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:57:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:57:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:57:57.447423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32923
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:57:57.533941 17400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:57:57.534188 17400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:57:57.534214 17400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:57:57.535569 17400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:57:57.535629 17400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:57:57.535643 17400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:57:57.535656 17400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:57:57.535669 17400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:57:57.537292 17400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:57:57.537595 17400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:57:57.537854 17400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:57:57.543457 17405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:57.543497 17408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:57.543541 17400 server_base.cc:1034] running on GCE node
W20260928 04:57:57.543497 17406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:57.543968 17400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:57:57.544306 17400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:57:57.545498 17400 hybrid_clock.cc:648] HybridClock initialized: now 1790571477545477 us; error 31 us; skew 500 ppm
Sep 28 04:57:57 dist-test-slave-m39l krb5kdc[17387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:57:57.548732 17400 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:57:57.549333 17400 webserver.cc:460] Webserver started at http://127.0.98.62:45495/ using document root <none> and password file <none>
I20260928 04:57:57.549605 17400 fs_manager.cc:359] Metadata directory not provided
I20260928 04:57:57.549687 17400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:57:57.549861 17400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:57:57.551272 17400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "9e8fd1b662104f278afee46e40e82b75"
format_stamp: "Formatted at 2026-09-28 04:57:57 on dist-test-slave-m39l"
server_key: "d706f88b3d6d7ea837873373b46a901d"
server_key_iv: "7de57cb50cb163d0af8908a49c1751f3"
server_key_version: "encryptionkey@0"
I20260928 04:57:57.551713 17400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "9e8fd1b662104f278afee46e40e82b75"
format_stamp: "Formatted at 2026-09-28 04:57:57 on dist-test-slave-m39l"
server_key: "d706f88b3d6d7ea837873373b46a901d"
server_key_iv: "7de57cb50cb163d0af8908a49c1751f3"
server_key_version: "encryptionkey@0"
I20260928 04:57:57.553357 17400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:57:57.554328 17415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:57.554606 17400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:57:57.554724 17400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "9e8fd1b662104f278afee46e40e82b75"
format_stamp: "Formatted at 2026-09-28 04:57:57 on dist-test-slave-m39l"
server_key: "d706f88b3d6d7ea837873373b46a901d"
server_key_iv: "7de57cb50cb163d0af8908a49c1751f3"
server_key_version: "encryptionkey@0"
I20260928 04:57:57.554819 17400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:57:57.584358 17400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:57:57.586898 17400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:57:57.587085 17400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:57:57.591640 17400 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32923
I20260928 04:57:57.591774 17467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32923 every 8 connection(s)
I20260928 04:57:57.592113 17400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:57:57.593143 17468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:57:57.593297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 17400
I20260928 04:57:57.593422 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:57:57.593667 392 external_mini_cluster.cc:1428] Setting key fd2cd2a1174754821dad19599e40ba37
I20260928 04:57:57.596997 17468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75: Bootstrap starting.
I20260928 04:57:57.597819 17468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75: Neither blocks nor log segments found. Creating new log.
I20260928 04:57:57.598133 17468 log.cc:824] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75: Log is configured to *not* fsync() on all Append() calls
I20260928 04:57:57.599104 17468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75: No bootstrap required, opened a new log
Sep 28 04:57:57 dist-test-slave-m39l krb5kdc[17387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:57:57.601728 17468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8fd1b662104f278afee46e40e82b75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32923 } }
I20260928 04:57:57.601881 17468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:57:57.601904 17468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8fd1b662104f278afee46e40e82b75, State: Initialized, Role: FOLLOWER
I20260928 04:57:57.602046 17468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8fd1b662104f278afee46e40e82b75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32923 } }
I20260928 04:57:57.602134 17468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:57:57.602195 17468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:57:57.602249 17468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:57:57.603093 17468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8fd1b662104f278afee46e40e82b75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32923 } }
I20260928 04:57:57.603219 17468 leader_election.cc:302] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e8fd1b662104f278afee46e40e82b75; no voters:
I20260928 04:57:57.603444 17468 leader_election.cc:288] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:57:57.603477 17473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:57:57.603596 17473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75 [term 1 LEADER]: Becoming Leader. State: Replica: 9e8fd1b662104f278afee46e40e82b75, State: Running, Role: LEADER
I20260928 04:57:57.603757 17468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:57:57.603742 17473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8fd1b662104f278afee46e40e82b75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32923 } }
I20260928 04:57:57.604249 17474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e8fd1b662104f278afee46e40e82b75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8fd1b662104f278afee46e40e82b75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32923 } } }
I20260928 04:57:57.604269 17475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e8fd1b662104f278afee46e40e82b75. Latest consensus state: current_term: 1 leader_uuid: "9e8fd1b662104f278afee46e40e82b75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8fd1b662104f278afee46e40e82b75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32923 } } }
I20260928 04:57:57.604449 17475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75 [sys.catalog]: This master's current role is: LEADER
I20260928 04:57:57.604703 17474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75 [sys.catalog]: This master's current role is: LEADER
I20260928 04:57:57.604717 17482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:57:57.605664 17482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:57:57.606338 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:57.594282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.0.98.62:32923)
0928 04:57:57.594469 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:57.594529 (+ 60us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:57.594574 (+ 45us) server_negotiation.cc:407] Connection header received
0928 04:57:57.595037 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:57.595044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:57.595114 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:57.595263 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:57:57.596199 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:57.597191 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:57.598319 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:57.599023 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:57.599134 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:57.602078 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:57:57.602083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:57:57.602084 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:57:57.602099 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:57:57.604217 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:57.604697 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:57.604701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:57.604701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:57.604876 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:57.605142 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:57.605143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:57.605143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:57.605475 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:57:57.605634 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:57.605912 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 04:57:57.606074 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260928 04:57:57.607496 17482 catalog_manager.cc:1347] Generated new cluster ID: 83bc3e37d3a7419ca5f662095ec64020
I20260928 04:57:57.607556 17482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:57:57.632256 17482 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:57:57.632835 17482 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:57:57.641848 17482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75: Generated new TSK 0
I20260928 04:57:57.642087 17482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:57:57.678565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32923
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:57:57.778294 17494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:57:57.778625 17494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:57:57.778666 17494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:57:57.780086 17494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:57:57.780158 17494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:57:57.780192 17494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:57:57.781962 17494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:57:57.782282 17494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:57:57.782573 17494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:57:57.788040 17499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:57.788040 17502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:57.788095 17500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:57.788081 17494 server_base.cc:1034] running on GCE node
I20260928 04:57:57.788545 17494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:57:57.788833 17494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:57:57.790014 17494 hybrid_clock.cc:648] HybridClock initialized: now 1790571477789957 us; error 76 us; skew 500 ppm
Sep 28 04:57:57 dist-test-slave-m39l krb5kdc[17387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:57:57.793087 17494 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:57:57.793599 17494 webserver.cc:460] Webserver started at http://127.0.98.1:36465/ using document root <none> and password file <none>
I20260928 04:57:57.793851 17494 fs_manager.cc:359] Metadata directory not provided
I20260928 04:57:57.793905 17494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:57:57.794107 17494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:57:57.795346 17494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "9f9e3075f71347018a76f958a0db4f9f"
format_stamp: "Formatted at 2026-09-28 04:57:57 on dist-test-slave-m39l"
server_key: "6e6f57c8ce471091ea1616c95c462aab"
server_key_iv: "4549c21562298cdccec0e9fab656f71b"
server_key_version: "encryptionkey@0"
I20260928 04:57:57.795746 17494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9f9e3075f71347018a76f958a0db4f9f"
format_stamp: "Formatted at 2026-09-28 04:57:57 on dist-test-slave-m39l"
server_key: "6e6f57c8ce471091ea1616c95c462aab"
server_key_iv: "4549c21562298cdccec0e9fab656f71b"
server_key_version: "encryptionkey@0"
I20260928 04:57:57.797338 17494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:57:57.798286 17509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:57.798575 17494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:57:57.798702 17494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "9f9e3075f71347018a76f958a0db4f9f"
format_stamp: "Formatted at 2026-09-28 04:57:57 on dist-test-slave-m39l"
server_key: "6e6f57c8ce471091ea1616c95c462aab"
server_key_iv: "4549c21562298cdccec0e9fab656f71b"
server_key_version: "encryptionkey@0"
I20260928 04:57:57.798785 17494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:57:57.812285 17494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:57:57.814883 17494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:57:57.815006 17494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:57:57.815249 17494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:57:57.815596 17494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:57:57.815655 17494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:57.815724 17494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:57:57.815755 17494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:57.822038 17494 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37165
I20260928 04:57:57.822187 17622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37165 every 8 connection(s)
I20260928 04:57:57.822549 17494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:57:57.824713 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 17494
I20260928 04:57:57.824841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:57:57.825064 392 external_mini_cluster.cc:1428] Setting key 44457de2e46d3abbc03c3ce3766c0081
Sep 28 04:57:57 dist-test-slave-m39l krb5kdc[17387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 04:57:57.834421 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:57.823460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58999 (local address 127.0.98.62:32923)
0928 04:57:57.823556 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:57.823558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:57.823673 (+ 115us) server_negotiation.cc:407] Connection header received
0928 04:57:57.824790 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:57.824792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:57.824812 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:57.824875 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:57:57.825799 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:57.826586 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:57.828065 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:57.828280 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:57.828340 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:57.830944 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:57:57.830946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:57:57.830947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:57:57.830955 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:57:57.832390 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:57.832834 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:57.832835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:57.832836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:57.832904 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:57.833273 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:57.833274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:57.833275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:57.833472 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:57:57.833561 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:57.834010 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 04:57:57.834258 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 04:57:57.834436 17625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:57.823376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32923 (local address 127.0.98.1:58999)
0928 04:57:57.823533 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:57.823597 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:57.824665 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:57.824951 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:57.824957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:57.825395 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:57:57.825696 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:57.825703 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:57.826680 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:57.826681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:57.827969 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:57.827973 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:57.828373 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:57.828374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:57.828526 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:57.829082 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:57:57.829092 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:57:57.830836 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:57:57.832487 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:57:57.832496 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:57:57.832498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:57:57.832749 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:57:57.833007 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:57:57.833008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:57:57.833009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:57:57.833160 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:57:57.833564 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:57:57.833572 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:57:57.833813 (+ 241us) client_negotiation.cc:769] Sending connection context
0928 04:57:57.833957 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 04:57:57.834178 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260928 04:57:57.835071 17623 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32923
I20260928 04:57:57.835203 17623 heartbeater.cc:464] Registering TS with master...
I20260928 04:57:57.835500 17623 heartbeater.cc:511] Master 127.0.98.62:32923 requested a full tablet report, sending...
I20260928 04:57:57.836129 17432 ts_manager.cc:194] Registered new tserver with Master: 9f9e3075f71347018a76f958a0db4f9f (127.0.98.1:37165)
I20260928 04:57:57.836807 17432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:57:57.854002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32923
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:57:57.938459 17628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:57:57.938767 17628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:57:57.938812 17628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:57:57.940038 17628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:57:57.940084 17628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:57:57.940121 17628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:57:57.941617 17628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:57:57.941828 17628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:57:57.942022 17628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:57:57.947265 17634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:57.947245 17633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:57.947245 17636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:57.947358 17628 server_base.cc:1034] running on GCE node
I20260928 04:57:57.947700 17628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:57:57.947969 17628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:57:57.949143 17628 hybrid_clock.cc:648] HybridClock initialized: now 1790571477949119 us; error 38 us; skew 500 ppm
Sep 28 04:57:57 dist-test-slave-m39l krb5kdc[17387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:57:57.952018 17628 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:57:57.952601 17628 webserver.cc:460] Webserver started at http://127.0.98.2:43153/ using document root <none> and password file <none>
I20260928 04:57:57.952850 17628 fs_manager.cc:359] Metadata directory not provided
I20260928 04:57:57.952924 17628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:57:57.953078 17628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:57:57.954183 17628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "2cb7a0e61af94080821bb6f8af9caceb"
format_stamp: "Formatted at 2026-09-28 04:57:57 on dist-test-slave-m39l"
server_key: "f3c478615ff9430d050ee14409fcc9ad"
server_key_iv: "7d3fe2e201e833ff1d1e942642cce286"
server_key_version: "encryptionkey@0"
I20260928 04:57:57.954634 17628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2cb7a0e61af94080821bb6f8af9caceb"
format_stamp: "Formatted at 2026-09-28 04:57:57 on dist-test-slave-m39l"
server_key: "f3c478615ff9430d050ee14409fcc9ad"
server_key_iv: "7d3fe2e201e833ff1d1e942642cce286"
server_key_version: "encryptionkey@0"
I20260928 04:57:57.956183 17628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:57:57.957100 17643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:57.957352 17628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:57:57.957434 17628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "2cb7a0e61af94080821bb6f8af9caceb"
format_stamp: "Formatted at 2026-09-28 04:57:57 on dist-test-slave-m39l"
server_key: "f3c478615ff9430d050ee14409fcc9ad"
server_key_iv: "7d3fe2e201e833ff1d1e942642cce286"
server_key_version: "encryptionkey@0"
I20260928 04:57:57.957530 17628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:57:57.967444 17628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:57:57.969866 17628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:57:57.969995 17628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:57:57.970196 17628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:57:57.970599 17628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:57:57.970639 17628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:57.970711 17628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:57:57.970741 17628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:57.977042 17628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41769
I20260928 04:57:57.977095 17756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41769 every 8 connection(s)
I20260928 04:57:57.977447 17628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:57:57.979547 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 17628
I20260928 04:57:57.979656 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:57:57.979882 392 external_mini_cluster.cc:1428] Setting key d9ee524b75d369272f24cb6e23d6e387
Sep 28 04:57:57 dist-test-slave-m39l krb5kdc[17387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:57:57.989907 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:57.978281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46219 (local address 127.0.98.62:32923)
0928 04:57:57.978343 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:57.978347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:57.978535 (+ 188us) server_negotiation.cc:407] Connection header received
0928 04:57:57.979703 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:57.979705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:57.979720 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:57.979810 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:57:57.980668 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:57.981171 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:57.982841 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:57.983222 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:57.983326 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:57.986256 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:57:57.986257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:57:57.986258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:57:57.986266 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:57:57.987802 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:57.988471 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:57.988472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:57.988473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:57.988549 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:57.988881 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:57.988882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:57.988882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:57.989040 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:57:57.989116 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:57.989645 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 04:57:57.989797 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 04:57:57.990136 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:57.978230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32923 (local address 127.0.98.2:46219)
0928 04:57:57.978392 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:57.978436 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:57.979525 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:57.979900 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:57.979905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:57.980305 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:57:57.980570 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:57.980576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:57.981290 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:57.981291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:57.982721 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:57.982724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:57.983329 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:57.983330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:57.983441 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:57.984070 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:57:57.984079 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:57:57.986144 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:57:57.987902 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:57:57.987912 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:57:57.987915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:57:57.988332 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:57:57.988639 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:57:57.988639 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:57:57.988640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:57:57.988796 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:57:57.989146 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:57:57.989154 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:57:57.989424 (+ 270us) client_negotiation.cc:769] Sending connection context
0928 04:57:57.989572 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 04:57:57.989840 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260928 04:57:57.990849 17757 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32923
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 04:57:57.990983 17757 heartbeater.cc:464] Registering TS with master...
I20260928 04:57:57.991220 17757 heartbeater.cc:511] Master 127.0.98.62:32923 requested a full tablet report, sending...
I20260928 04:57:57.991735 17432 ts_manager.cc:194] Registered new tserver with Master: 2cb7a0e61af94080821bb6f8af9caceb (127.0.98.2:41769)
I20260928 04:57:57.992223 17432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:57:58.013713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32923
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:57:58.100471 17762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:57:58.100706 17762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:57:58.100734 17762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:57:58.102044 17762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:57:58.102097 17762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:57:58.102126 17762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:57:58.103842 17762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:57:58.104146 17762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:57:58.104390 17762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:57:58.110004 17762 server_base.cc:1034] running on GCE node
W20260928 04:57:58.110077 17768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:58.110174 17770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:58.110004 17767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:58.110524 17762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:57:58.110819 17762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:57:58.111980 17762 hybrid_clock.cc:648] HybridClock initialized: now 1790571478111963 us; error 32 us; skew 500 ppm
Sep 28 04:57:58 dist-test-slave-m39l krb5kdc[17387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:57:58.114952 17762 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:57:58.115468 17762 webserver.cc:460] Webserver started at http://127.0.98.3:33111/ using document root <none> and password file <none>
I20260928 04:57:58.115716 17762 fs_manager.cc:359] Metadata directory not provided
I20260928 04:57:58.115790 17762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:57:58.115949 17762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:57:58.117082 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "57b6d7ac122d46db85e09ccc913d6676"
format_stamp: "Formatted at 2026-09-28 04:57:58 on dist-test-slave-m39l"
server_key: "b8bbca61ab47a6eba61ea07e2122d4de"
server_key_iv: "5837edbff0137da625192bbe4a85894e"
server_key_version: "encryptionkey@0"
I20260928 04:57:58.117496 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "57b6d7ac122d46db85e09ccc913d6676"
format_stamp: "Formatted at 2026-09-28 04:57:58 on dist-test-slave-m39l"
server_key: "b8bbca61ab47a6eba61ea07e2122d4de"
server_key_iv: "5837edbff0137da625192bbe4a85894e"
server_key_version: "encryptionkey@0"
I20260928 04:57:58.119012 17762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:57:58.119946 17777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:58.120157 17762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:57:58.120282 17762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "57b6d7ac122d46db85e09ccc913d6676"
format_stamp: "Formatted at 2026-09-28 04:57:58 on dist-test-slave-m39l"
server_key: "b8bbca61ab47a6eba61ea07e2122d4de"
server_key_iv: "5837edbff0137da625192bbe4a85894e"
server_key_version: "encryptionkey@0"
I20260928 04:57:58.120376 17762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:57:58.149143 17762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:57:58.151510 17762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:57:58.151700 17762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:57:58.151957 17762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:57:58.152283 17762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:57:58.152338 17762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:58.152374 17762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:57:58.152420 17762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:58.158964 17762 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42237
I20260928 04:57:58.159045 17890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42237 every 8 connection(s)
I20260928 04:57:58.159390 17762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:57:58 dist-test-slave-m39l krb5kdc[17387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:57:58.169462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 17762
I20260928 04:57:58.169572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:57:58.169577 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:58.160231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58469 (local address 127.0.98.62:32923)
0928 04:57:58.160308 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:58.160310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:58.160587 (+ 277us) server_negotiation.cc:407] Connection header received
0928 04:57:58.161654 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:58.161655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:58.161667 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:58.161734 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:57:58.162392 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.162975 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.163985 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.164191 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.164262 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:58.166460 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:57:58.166462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:57:58.166462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:57:58.166470 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:57:58.167987 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:58.168452 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:58.168452 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:58.168453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:58.168507 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:58.168787 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:58.168789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:58.168789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:58.169006 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:57:58.169085 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:58.169365 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 04:57:58.169466 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 04:57:58.169732 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:58.160128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32923 (local address 127.0.98.3:58469)
0928 04:57:58.160453 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:58.160499 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:58.161532 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:58.161819 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:58.161823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:58.162095 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:57:58.162298 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:58.162302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.163058 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.163059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:58.163897 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:58.163900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.164282 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.164283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:58.164386 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:58.164913 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:57:58.164922 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:57:58.166363 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:57:58.168077 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:57:58.168085 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:57:58.168087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:57:58.168379 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:57:58.168565 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:57:58.168565 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:57:58.168566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:57:58.168695 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:57:58.169080 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:57:58.169085 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:57:58.169218 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 04:57:58.169304 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 04:57:58.169436 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":256,"threads_started":1}
I20260928 04:57:58.169811 392 external_mini_cluster.cc:1428] Setting key 9291e04b816d8cc18c348a540b08fef4
I20260928 04:57:58.170367 17891 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32923
I20260928 04:57:58.170480 17891 heartbeater.cc:464] Registering TS with master...
I20260928 04:57:58.170709 17891 heartbeater.cc:511] Master 127.0.98.62:32923 requested a full tablet report, sending...
I20260928 04:57:58.171103 17432 ts_manager.cc:194] Registered new tserver with Master: 57b6d7ac122d46db85e09ccc913d6676 (127.0.98.3:42237)
I20260928 04:57:58.171454 17432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58469
I20260928 04:57:58.182236 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:57:58.189913 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:58.183710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45814 (local address 127.0.98.62:32923)
0928 04:57:58.183774 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:58.183776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:58.183884 (+ 108us) server_negotiation.cc:407] Connection header received
0928 04:57:58.183931 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:58.183932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:58.183941 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:58.183995 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:57:58.184614 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.185224 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.185954 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.186134 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.186206 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:58.187515 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:57:58.187516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:57:58.187516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:57:58.187523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:57:58.188819 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:58.189220 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:58.189220 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:58.189221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:58.189292 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:57:58.189445 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:57:58.189446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:57:58.189446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:57:58.189595 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:57:58.189671 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:58.189780 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 04:57:58.189844 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:57:58.191737 17432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:57:58.193380 17432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:57:58.200346 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:58.196262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47752 (local address 127.0.98.1:37165)
0928 04:57:58.196432 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:58.196438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:58.196553 (+ 115us) server_negotiation.cc:407] Connection header received
0928 04:57:58.196605 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:58.196608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:58.196625 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:58.196713 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:57:58.196906 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.197802 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.199233 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.199923 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.200016 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:58.200089 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:58.200225 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 04:57:58.200268 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260928 04:57:58.200341 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:58.195967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37165 (local address 127.0.0.1:47752)
0928 04:57:58.196485 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:58.196493 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:58.196543 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:58.196729 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:58.196733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:58.196735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:57:58.196816 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:58.196819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.197916 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.197917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:58.199135 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:58.199138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.200016 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.200017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:58.200104 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:58.200118 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 04:57:58.200194 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 04:57:58.200255 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":8,"thread_start_us":96,"threads_started":1}
I20260928 04:57:58.200675 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:58.195967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41769 (local address 127.0.0.1:35972)
0928 04:57:58.196261 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:58.196309 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:58.196380 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:58.197130 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:58.197131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:58.197133 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:57:58.197186 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:58.197187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.198453 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.198454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:58.199549 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:58.199552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.200435 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.200435 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:58.200494 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:58.200501 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 04:57:58.200532 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 04:57:58.200584 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":103,"threads_started":1}
I20260928 04:57:58.200675 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:58.195957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42237 (local address 127.0.0.1:49082)
0928 04:57:58.196261 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:58.196309 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:58.196380 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:58.196953 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:58.196954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:58.196963 (+ 9us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:57:58.197007 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:58.197008 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.198382 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.198383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:58.199410 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:58.199412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.200425 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.200426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:58.200494 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:58.200506 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 04:57:58.200561 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 04:57:58.200596 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":4,"thread_start_us":64,"threads_started":1}
I20260928 04:57:58.200829 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:58.196041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.0.98.3:42237)
0928 04:57:58.196691 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:58.196696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:58.196707 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:57:58.196749 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:58.196751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:58.196769 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:58.196847 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:57:58.197080 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.198255 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.199506 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.200254 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.200331 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:58.200390 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:58.200602 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 04:57:58.200659 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":141,"threads_started":1}
I20260928 04:57:58.201033 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:58.196475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35972 (local address 127.0.98.2:41769)
0928 04:57:58.196880 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:58.196884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:58.196893 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:57:58.196987 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:58.196990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:58.197004 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:58.197257 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:57:58.197343 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.198376 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.199644 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.200344 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.200810 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:58.200880 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:58.200911 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 04:57:58.200957 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":111,"threads_started":1}
I20260928 04:57:58.201614 17825 tablet_service.cc:1467] Processing CreateTablet for tablet a5386fc275c84b928738a5428ca194aa (DEFAULT_TABLE table=test-table [id=89ba4c1570f547e4ab8716a5c7639f1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:57:58.202034 17825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5386fc275c84b928738a5428ca194aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:57:58.202128 17557 tablet_service.cc:1467] Processing CreateTablet for tablet a5386fc275c84b928738a5428ca194aa (DEFAULT_TABLE table=test-table [id=89ba4c1570f547e4ab8716a5c7639f1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:57:58.202450 17557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5386fc275c84b928738a5428ca194aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:57:58.205766 17909 tablet_bootstrap.cc:492] T a5386fc275c84b928738a5428ca194aa P 9f9e3075f71347018a76f958a0db4f9f: Bootstrap starting.
I20260928 04:57:58.206070 17908 tablet_bootstrap.cc:492] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676: Bootstrap starting.
I20260928 04:57:58.206359 17691 tablet_service.cc:1467] Processing CreateTablet for tablet a5386fc275c84b928738a5428ca194aa (DEFAULT_TABLE table=test-table [id=89ba4c1570f547e4ab8716a5c7639f1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:57:58.206573 17909 tablet_bootstrap.cc:654] T a5386fc275c84b928738a5428ca194aa P 9f9e3075f71347018a76f958a0db4f9f: Neither blocks nor log segments found. Creating new log.
I20260928 04:57:58.206724 17691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5386fc275c84b928738a5428ca194aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:57:58.206770 17908 tablet_bootstrap.cc:654] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676: Neither blocks nor log segments found. Creating new log.
I20260928 04:57:58.207230 17909 log.cc:824] T a5386fc275c84b928738a5428ca194aa P 9f9e3075f71347018a76f958a0db4f9f: Log is configured to *not* fsync() on all Append() calls
I20260928 04:57:58.207419 17908 log.cc:824] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676: Log is configured to *not* fsync() on all Append() calls
I20260928 04:57:58.208081 17909 tablet_bootstrap.cc:492] T a5386fc275c84b928738a5428ca194aa P 9f9e3075f71347018a76f958a0db4f9f: No bootstrap required, opened a new log
I20260928 04:57:58.208094 17908 tablet_bootstrap.cc:492] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676: No bootstrap required, opened a new log
I20260928 04:57:58.208163 17909 ts_tablet_manager.cc:1397] T a5386fc275c84b928738a5428ca194aa P 9f9e3075f71347018a76f958a0db4f9f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:57:58.208195 17908 ts_tablet_manager.cc:1397] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 04:57:58.209802 17908 raft_consensus.cc:348] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9e3075f71347018a76f958a0db4f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37165 } } peers { permanent_uuid: "57b6d7ac122d46db85e09ccc913d6676" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42237 } } peers { permanent_uuid: "2cb7a0e61af94080821bb6f8af9caceb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } }
I20260928 04:57:58.209973 17908 raft_consensus.cc:374] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:57:58.210016 17908 raft_consensus.cc:729] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b6d7ac122d46db85e09ccc913d6676, State: Initialized, Role: FOLLOWER
I20260928 04:57:58.210147 17908 consensus_queue.cc:260] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9e3075f71347018a76f958a0db4f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37165 } } peers { permanent_uuid: "57b6d7ac122d46db85e09ccc913d6676" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42237 } } peers { permanent_uuid: "2cb7a0e61af94080821bb6f8af9caceb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } }
I20260928 04:57:58.210331 17909 raft_consensus.cc:348] T a5386fc275c84b928738a5428ca194aa P 9f9e3075f71347018a76f958a0db4f9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9e3075f71347018a76f958a0db4f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37165 } } peers { permanent_uuid: "57b6d7ac122d46db85e09ccc913d6676" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42237 } } peers { permanent_uuid: "2cb7a0e61af94080821bb6f8af9caceb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } }
I20260928 04:57:58.210469 17891 heartbeater.cc:503] Master 127.0.98.62:32923 was elected leader, sending a full tablet report...
I20260928 04:57:58.210557 17909 raft_consensus.cc:374] T a5386fc275c84b928738a5428ca194aa P 9f9e3075f71347018a76f958a0db4f9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:57:58.210614 17909 raft_consensus.cc:729] T a5386fc275c84b928738a5428ca194aa P 9f9e3075f71347018a76f958a0db4f9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9e3075f71347018a76f958a0db4f9f, State: Initialized, Role: FOLLOWER
I20260928 04:57:58.210747 17909 consensus_queue.cc:260] T a5386fc275c84b928738a5428ca194aa P 9f9e3075f71347018a76f958a0db4f9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9e3075f71347018a76f958a0db4f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37165 } } peers { permanent_uuid: "57b6d7ac122d46db85e09ccc913d6676" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42237 } } peers { permanent_uuid: "2cb7a0e61af94080821bb6f8af9caceb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } }
I20260928 04:57:58.210840 17912 tablet_bootstrap.cc:492] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb: Bootstrap starting.
I20260928 04:57:58.211038 17909 ts_tablet_manager.cc:1428] T a5386fc275c84b928738a5428ca194aa P 9f9e3075f71347018a76f958a0db4f9f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 04:57:58.211227 17623 heartbeater.cc:503] Master 127.0.98.62:32923 was elected leader, sending a full tablet report...
I20260928 04:57:58.211304 17908 ts_tablet_manager.cc:1428] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:57:58.211536 17912 tablet_bootstrap.cc:654] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb: Neither blocks nor log segments found. Creating new log.
I20260928 04:57:58.212157 17912 log.cc:824] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb: Log is configured to *not* fsync() on all Append() calls
I20260928 04:57:58.212875 17912 tablet_bootstrap.cc:492] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb: No bootstrap required, opened a new log
I20260928 04:57:58.212930 17912 ts_tablet_manager.cc:1397] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:57:58.214322 17912 raft_consensus.cc:348] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9e3075f71347018a76f958a0db4f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37165 } } peers { permanent_uuid: "57b6d7ac122d46db85e09ccc913d6676" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42237 } } peers { permanent_uuid: "2cb7a0e61af94080821bb6f8af9caceb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } }
I20260928 04:57:58.214463 17912 raft_consensus.cc:374] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:57:58.214555 17912 raft_consensus.cc:729] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb7a0e61af94080821bb6f8af9caceb, State: Initialized, Role: FOLLOWER
I20260928 04:57:58.214695 17912 consensus_queue.cc:260] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9e3075f71347018a76f958a0db4f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37165 } } peers { permanent_uuid: "57b6d7ac122d46db85e09ccc913d6676" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42237 } } peers { permanent_uuid: "2cb7a0e61af94080821bb6f8af9caceb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } }
I20260928 04:57:58.214998 17757 heartbeater.cc:503] Master 127.0.98.62:32923 was elected leader, sending a full tablet report...
I20260928 04:57:58.215004 17912 ts_tablet_manager.cc:1428] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 04:57:58.228022 17758 tablet.cc:2369] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:57:58.323370 17624 tablet.cc:2369] T a5386fc275c84b928738a5428ca194aa P 9f9e3075f71347018a76f958a0db4f9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:57:58.410596 17892 tablet.cc:2369] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:57:58.503301 17913 raft_consensus.cc:482] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:57:58.503465 17913 raft_consensus.cc:504] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9e3075f71347018a76f958a0db4f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37165 } } peers { permanent_uuid: "57b6d7ac122d46db85e09ccc913d6676" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42237 } } peers { permanent_uuid: "2cb7a0e61af94080821bb6f8af9caceb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } }
I20260928 04:57:58.503906 17913 leader_election.cc:288] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f9e3075f71347018a76f958a0db4f9f (127.0.98.1:37165), 2cb7a0e61af94080821bb6f8af9caceb (127.0.98.2:41769)
I20260928 04:57:58.507227 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:58.504128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37165 (local address 127.0.98.3:39679)
0928 04:57:58.504187 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:58.504199 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:58.504253 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:58.504448 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:58.504449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:58.504451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:57:58.504535 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:58.504538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.505420 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.505421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:58.506297 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:58.506300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.506953 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.506954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:58.507041 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:58.507044 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:57:58.507100 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 04:57:58.507140 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":21}
I20260928 04:57:58.507256 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:58.504197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39679 (local address 127.0.98.1:37165)
0928 04:57:58.504241 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:58.504244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:58.504255 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:57:58.504350 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:58.504351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:58.504364 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:58.504423 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:57:58.504630 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.505337 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.506404 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.506868 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.506928 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:58.506946 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:58.507119 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 04:57:58.507163 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:57:58.507369 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:58.504130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41769 (local address 127.0.98.3:39519)
0928 04:57:58.504327 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:58.504338 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:58.504395 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:58.504607 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:58.504608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:58.504611 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:57:58.504692 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:58.504694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.505480 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.505480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:58.506453 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:58.506456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.507052 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.507053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:58.507163 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:58.507167 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:57:58.507235 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 04:57:58.507276 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":17,"thread_start_us":119,"threads_started":1}
I20260928 04:57:58.507391 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:58.504251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39519 (local address 127.0.98.2:41769)
0928 04:57:58.504300 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:58.504302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:58.504451 (+ 149us) server_negotiation.cc:407] Connection header received
0928 04:57:58.504481 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:58.504482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:58.504496 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:58.504556 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:57:58.504819 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.505367 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.506570 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.506962 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.507034 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:58.507044 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:58.507260 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 04:57:58.507308 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:57:58.507577 17577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5386fc275c84b928738a5428ca194aa" candidate_uuid: "57b6d7ac122d46db85e09ccc913d6676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9e3075f71347018a76f958a0db4f9f" is_pre_election: true
I20260928 04:57:58.507755 17577 raft_consensus.cc:2454] T a5386fc275c84b928738a5428ca194aa P 9f9e3075f71347018a76f958a0db4f9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57b6d7ac122d46db85e09ccc913d6676 in term 0.
I20260928 04:57:58.507779 17711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5386fc275c84b928738a5428ca194aa" candidate_uuid: "57b6d7ac122d46db85e09ccc913d6676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb7a0e61af94080821bb6f8af9caceb" is_pre_election: true
I20260928 04:57:58.507946 17711 raft_consensus.cc:2454] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57b6d7ac122d46db85e09ccc913d6676 in term 0.
I20260928 04:57:58.508018 17779 leader_election.cc:302] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57b6d7ac122d46db85e09ccc913d6676, 9f9e3075f71347018a76f958a0db4f9f; no voters:
I20260928 04:57:58.508195 17913 raft_consensus.cc:2781] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:57:58.508240 17913 raft_consensus.cc:482] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:57:58.508256 17913 raft_consensus.cc:3037] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:57:58.508993 17913 raft_consensus.cc:504] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9e3075f71347018a76f958a0db4f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37165 } } peers { permanent_uuid: "57b6d7ac122d46db85e09ccc913d6676" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42237 } } peers { permanent_uuid: "2cb7a0e61af94080821bb6f8af9caceb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } }
I20260928 04:57:58.509141 17913 leader_election.cc:288] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [CANDIDATE]: Term 1 election: Requested vote from peers 9f9e3075f71347018a76f958a0db4f9f (127.0.98.1:37165), 2cb7a0e61af94080821bb6f8af9caceb (127.0.98.2:41769)
I20260928 04:57:58.509322 17711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5386fc275c84b928738a5428ca194aa" candidate_uuid: "57b6d7ac122d46db85e09ccc913d6676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb7a0e61af94080821bb6f8af9caceb"
I20260928 04:57:58.509313 17577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5386fc275c84b928738a5428ca194aa" candidate_uuid: "57b6d7ac122d46db85e09ccc913d6676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9e3075f71347018a76f958a0db4f9f"
I20260928 04:57:58.509402 17711 raft_consensus.cc:3037] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:57:58.509402 17577 raft_consensus.cc:3037] T a5386fc275c84b928738a5428ca194aa P 9f9e3075f71347018a76f958a0db4f9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:57:58.510269 17577 raft_consensus.cc:2454] T a5386fc275c84b928738a5428ca194aa P 9f9e3075f71347018a76f958a0db4f9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57b6d7ac122d46db85e09ccc913d6676 in term 1.
I20260928 04:57:58.510269 17711 raft_consensus.cc:2454] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57b6d7ac122d46db85e09ccc913d6676 in term 1.
I20260928 04:57:58.510543 17779 leader_election.cc:302] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57b6d7ac122d46db85e09ccc913d6676, 9f9e3075f71347018a76f958a0db4f9f; no voters:
I20260928 04:57:58.510656 17913 raft_consensus.cc:2781] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:57:58.511353 17913 raft_consensus.cc:686] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [term 1 LEADER]: Becoming Leader. State: Replica: 57b6d7ac122d46db85e09ccc913d6676, State: Running, Role: LEADER
I20260928 04:57:58.511482 17913 consensus_queue.cc:237] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9e3075f71347018a76f958a0db4f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37165 } } peers { permanent_uuid: "57b6d7ac122d46db85e09ccc913d6676" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42237 } } peers { permanent_uuid: "2cb7a0e61af94080821bb6f8af9caceb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } }
I20260928 04:57:58.512447 17432 catalog_manager.cc:5495] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57b6d7ac122d46db85e09ccc913d6676 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "57b6d7ac122d46db85e09ccc913d6676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9e3075f71347018a76f958a0db4f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57b6d7ac122d46db85e09ccc913d6676" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cb7a0e61af94080821bb6f8af9caceb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } health_report { overall_health: UNKNOWN } } }
I20260928 04:57:58.558444 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:58.555611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.0.98.3:42237)
0928 04:57:58.555683 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:58.555686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:58.555696 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:57:58.555745 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:58.555746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:58.555765 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:58.555855 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:57:58.556029 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.556515 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.557392 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.557653 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.557755 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:58.557989 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:57:58.558099 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:57:58.558274 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:58.558295 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 04:57:58.558316 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 04:57:58.560489 17711 raft_consensus.cc:1261] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb [term 1 FOLLOWER]: Refusing update from remote peer 57b6d7ac122d46db85e09ccc913d6676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:57:58.560527 17577 raft_consensus.cc:1261] T a5386fc275c84b928738a5428ca194aa P 9f9e3075f71347018a76f958a0db4f9f [term 1 FOLLOWER]: Refusing update from remote peer 57b6d7ac122d46db85e09ccc913d6676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:57:58.560967 17918 consensus_queue.cc:1035] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f9e3075f71347018a76f958a0db4f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:57:58.561159 17913 consensus_queue.cc:1035] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb7a0e61af94080821bb6f8af9caceb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:57:58.566561 17432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:57:58.566696 17432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:57:58.567461 17432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e8fd1b662104f278afee46e40e82b75: Sending DeleteTablet for 3 replicas of tablet a5386fc275c84b928738a5428ca194aa
I20260928 04:57:58.567817 17557 tablet_service.cc:1514] Processing DeleteTablet for tablet a5386fc275c84b928738a5428ca194aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:57:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47752
I20260928 04:57:58.567862 17691 tablet_service.cc:1514] Processing DeleteTablet for tablet a5386fc275c84b928738a5428ca194aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:57:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35972
I20260928 04:57:58.568092 17825 tablet_service.cc:1514] Processing DeleteTablet for tablet a5386fc275c84b928738a5428ca194aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:57:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49082
I20260928 04:57:58.568326 17933 tablet_replica.cc:330] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676: stopping tablet replica
I20260928 04:57:58.568491 17933 raft_consensus.cc:2229] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:57:58.568616 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 17494
I20260928 04:57:58.568645 17933 raft_consensus.cc:2258] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:57:58.570539 17933 ts_tablet_manager.cc:1905] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:57:58.570657 17932 tablet_replica.cc:330] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb: stopping tablet replica
I20260928 04:57:58.570847 17932 raft_consensus.cc:2229] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:57:58.570992 17932 raft_consensus.cc:2258] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:57:58.571499 17932 ts_tablet_manager.cc:1905] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:57:58.572569 17933 ts_tablet_manager.cc:1918] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:57:58.572685 17933 log.cc:1196] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/a5386fc275c84b928738a5428ca194aa
I20260928 04:57:58.572959 17933 ts_tablet_manager.cc:1939] T a5386fc275c84b928738a5428ca194aa P 57b6d7ac122d46db85e09ccc913d6676: Deleting consensus metadata
I20260928 04:57:58.573410 17418 catalog_manager.cc:4841] TS 57b6d7ac122d46db85e09ccc913d6676 (127.0.98.3:42237): tablet a5386fc275c84b928738a5428ca194aa (table test-table [id=89ba4c1570f547e4ab8716a5c7639f1e]) successfully deleted
I20260928 04:57:58.573457 17932 ts_tablet_manager.cc:1918] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:57:58.573534 17932 log.cc:1196] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/a5386fc275c84b928738a5428ca194aa
I20260928 04:57:58.573869 17932 ts_tablet_manager.cc:1939] T a5386fc275c84b928738a5428ca194aa P 2cb7a0e61af94080821bb6f8af9caceb: Deleting consensus metadata
I20260928 04:57:58.574334 17416 catalog_manager.cc:4841] TS 2cb7a0e61af94080821bb6f8af9caceb (127.0.98.2:41769): tablet a5386fc275c84b928738a5428ca194aa (table test-table [id=89ba4c1570f547e4ab8716a5c7639f1e]) successfully deleted
W20260928 04:57:58.577615 17417 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37165 (error 108)
I20260928 04:57:58.578253 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:58.578097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37165 (local address 127.0.0.1:47766)
0928 04:57:58.578168 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:58.578215 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 04:57:58.578296 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 17628
W20260928 04:57:58.585124 17417 catalog_manager.cc:4584] TS 9f9e3075f71347018a76f958a0db4f9f (127.0.98.1:37165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a5386fc275c84b928738a5428ca194aa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37165: connect: Connection refused (error 111)
I20260928 04:57:58.585214 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 17762
I20260928 04:57:58.590929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 17400
2026-09-28T04:57:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2322 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T04:57:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:57:58Z Disabled control of system clock
I20260928 04:57:58.620424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46003
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:57:58.709425 17938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:57:58.709692 17938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:57:58.709715 17938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:57:58.711120 17938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:57:58.711176 17938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:57:58.711189 17938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:57:58.711202 17938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:57:58.711215 17938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:57:58.712702 17938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46003
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:57:58.712987 17938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:57:58.713336 17938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:57:58.718967 17946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:58.719164 17938 server_base.cc:1034] running on GCE node
W20260928 04:57:58.718914 17943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:58.718914 17944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:58.719624 17938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:57:58.719995 17938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:57:58.721199 17938 hybrid_clock.cc:648] HybridClock initialized: now 1790571478721180 us; error 45 us; skew 500 ppm
I20260928 04:57:58.722645 17938 webserver.cc:460] Webserver started at http://127.0.98.62:33937/ using document root <none> and password file <none>
I20260928 04:57:58.722939 17938 fs_manager.cc:359] Metadata directory not provided
I20260928 04:57:58.723014 17938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:57:58.723207 17938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:57:58.724403 17938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "a7f347c6264e45b48fcc4236bde5eaba"
format_stamp: "Formatted at 2026-09-28 04:57:58 on dist-test-slave-m39l"
server_key: "d8634e6e2031909dcdeda1f950b6dcaa"
server_key_iv: "255b031c8716ad567ed7340578d83dfa"
server_key_version: "encryptionkey@0"
I20260928 04:57:58.724802 17938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "a7f347c6264e45b48fcc4236bde5eaba"
format_stamp: "Formatted at 2026-09-28 04:57:58 on dist-test-slave-m39l"
server_key: "d8634e6e2031909dcdeda1f950b6dcaa"
server_key_iv: "255b031c8716ad567ed7340578d83dfa"
server_key_version: "encryptionkey@0"
I20260928 04:57:58.726363 17938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:57:58.727320 17952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:58.727551 17938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:57:58.727667 17938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "a7f347c6264e45b48fcc4236bde5eaba"
format_stamp: "Formatted at 2026-09-28 04:57:58 on dist-test-slave-m39l"
server_key: "d8634e6e2031909dcdeda1f950b6dcaa"
server_key_iv: "255b031c8716ad567ed7340578d83dfa"
server_key_version: "encryptionkey@0"
I20260928 04:57:58.727752 17938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:57:58.736326 17938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:57:58.736709 17938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:57:58.736878 17938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:57:58.741147 17938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46003
I20260928 04:57:58.741196 18004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46003 every 8 connection(s)
I20260928 04:57:58.741607 17938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:57:58.742965 18005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:57:58.745411 18005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba: Bootstrap starting.
I20260928 04:57:58.745658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 17938
I20260928 04:57:58.745800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:57:58.746004 392 external_mini_cluster.cc:1428] Setting key f24964440a1bbab7e7c78bd37a9cf680
I20260928 04:57:58.746223 18005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba: Neither blocks nor log segments found. Creating new log.
I20260928 04:57:58.746639 18005 log.cc:824] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba: Log is configured to *not* fsync() on all Append() calls
I20260928 04:57:58.747474 18005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba: No bootstrap required, opened a new log
I20260928 04:57:58.749403 18005 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f347c6264e45b48fcc4236bde5eaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46003 } }
I20260928 04:57:58.749583 18005 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:57:58.749634 18005 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7f347c6264e45b48fcc4236bde5eaba, State: Initialized, Role: FOLLOWER
I20260928 04:57:58.749769 18005 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f347c6264e45b48fcc4236bde5eaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46003 } }
I20260928 04:57:58.749838 18005 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:57:58.749895 18005 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:57:58.749944 18005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:57:58.750612 18005 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f347c6264e45b48fcc4236bde5eaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46003 } }
I20260928 04:57:58.750751 18005 leader_election.cc:302] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7f347c6264e45b48fcc4236bde5eaba; no voters:
I20260928 04:57:58.750952 18005 leader_election.cc:288] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:57:58.751029 18010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:57:58.751230 18010 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba [term 1 LEADER]: Becoming Leader. State: Replica: a7f347c6264e45b48fcc4236bde5eaba, State: Running, Role: LEADER
I20260928 04:57:58.751253 18005 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:57:58.751329 18010 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f347c6264e45b48fcc4236bde5eaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46003 } }
I20260928 04:57:58.751659 18010 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f347c6264e45b48fcc4236bde5eaba. Latest consensus state: current_term: 1 leader_uuid: "a7f347c6264e45b48fcc4236bde5eaba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f347c6264e45b48fcc4236bde5eaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46003 } } }
I20260928 04:57:58.751830 18010 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba [sys.catalog]: This master's current role is: LEADER
I20260928 04:57:58.752214 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:58.746618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40508 (local address 127.0.98.62:46003)
0928 04:57:58.746947 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:58.746954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:58.746979 (+ 25us) server_negotiation.cc:407] Connection header received
0928 04:57:58.747408 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:58.747414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:58.747474 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:58.747693 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:57:58.747872 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.748790 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.749596 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.750188 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.750328 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:58.751159 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:57:58.751163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:57:58.751164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:57:58.751434 (+ 270us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:57:58.751476 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:57:58.751491 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:57:58.751558 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:58.751860 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 04:57:58.751979 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":97,"threads_started":1}
I20260928 04:57:58.752357 18019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:57:58.752866 18019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:57:58.752892 18011 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7f347c6264e45b48fcc4236bde5eaba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f347c6264e45b48fcc4236bde5eaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46003 } } }
I20260928 04:57:58.753315 18011 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba [sys.catalog]: This master's current role is: LEADER
I20260928 04:57:58.754724 18019 catalog_manager.cc:1347] Generated new cluster ID: fa4ab69bcacd459ea15f853c64c5b8f3
I20260928 04:57:58.754799 18019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:57:58.760133 18019 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:57:58.760747 18019 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:57:58.764539 18019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba: Generated new TSK 0
I20260928 04:57:58.764776 18019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:57:58.768045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46003
--builtin_ntp_servers=127.0.98.20:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:57:58.866168 18029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:57:58.866452 18029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:57:58.866492 18029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:57:58.868212 18029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:57:58.868328 18029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:57:58.868381 18029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:57:58.870100 18029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:57:58.870416 18029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:57:58.870781 18029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:57:58.877096 18034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:58.877096 18037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:58.877096 18035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:58.877444 18029 server_base.cc:1034] running on GCE node
I20260928 04:57:58.877674 18029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:57:58.877951 18029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:57:58.879144 18029 hybrid_clock.cc:648] HybridClock initialized: now 1790571478879128 us; error 33 us; skew 500 ppm
I20260928 04:57:58.880645 18029 webserver.cc:460] Webserver started at http://127.0.98.1:34391/ using document root <none> and password file <none>
I20260928 04:57:58.880908 18029 fs_manager.cc:359] Metadata directory not provided
I20260928 04:57:58.880981 18029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:57:58.881141 18029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:57:58.882315 18029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "8c0d3bcab0814e469f3bce736b07c4e7"
format_stamp: "Formatted at 2026-09-28 04:57:58 on dist-test-slave-m39l"
server_key: "d7d3be0194f06a10f49b363dd359cf41"
server_key_iv: "70a1ba8daf79dee96b879a2e82afd8de"
server_key_version: "encryptionkey@0"
I20260928 04:57:58.882718 18029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8c0d3bcab0814e469f3bce736b07c4e7"
format_stamp: "Formatted at 2026-09-28 04:57:58 on dist-test-slave-m39l"
server_key: "d7d3be0194f06a10f49b363dd359cf41"
server_key_iv: "70a1ba8daf79dee96b879a2e82afd8de"
server_key_version: "encryptionkey@0"
I20260928 04:57:58.884223 18029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:57:58.885039 18043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:58.885205 18029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:57:58.885320 18029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "8c0d3bcab0814e469f3bce736b07c4e7"
format_stamp: "Formatted at 2026-09-28 04:57:58 on dist-test-slave-m39l"
server_key: "d7d3be0194f06a10f49b363dd359cf41"
server_key_iv: "70a1ba8daf79dee96b879a2e82afd8de"
server_key_version: "encryptionkey@0"
I20260928 04:57:58.885402 18029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:57:58.903055 18029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:57:58.903427 18029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:57:58.903555 18029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:57:58.903861 18029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:57:58.904266 18029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:57:58.904304 18029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:58.904328 18029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:57:58.904340 18029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:58.910451 18029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42089
I20260928 04:57:58.910557 18156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42089 every 8 connection(s)
I20260928 04:57:58.910971 18029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:57:58.914155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 18029
I20260928 04:57:58.914278 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:57:58.914520 392 external_mini_cluster.cc:1428] Setting key fdf9942bbeda403adeb11c17f973e56b
I20260928 04:57:58.916152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46003
--builtin_ntp_servers=127.0.98.20:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 04:57:58.918512 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:58.911751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46003 (local address 127.0.98.1:37515)
0928 04:57:58.912006 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:58.912100 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:58.912678 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:58.912916 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:58.912920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:58.912929 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:57:58.913500 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:58.913505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.914453 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.914455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:58.915681 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:58.915685 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.916189 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.916201 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:58.916408 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:58.917517 (+ 1109us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:57:58.917518 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:57:58.917594 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:57:58.917636 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:57:58.917908 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:57:58.917917 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:57:58.917921 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:57:58.918012 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 04:57:58.918184 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":104,"threads_started":1}
I20260928 04:57:58.918803 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:58.911810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37515 (local address 127.0.98.62:46003)
0928 04:57:58.911897 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:58.911900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:58.912208 (+ 308us) server_negotiation.cc:407] Connection header received
0928 04:57:58.912781 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:58.912782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:58.912794 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:58.912852 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:57:58.913615 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.914298 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.915808 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:58.916012 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:58.916081 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:58.917748 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:57:58.917750 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:57:58.917750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:57:58.917756 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:57:58.917776 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:57:58.917781 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:57:58.917883 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:58.918654 (+ 771us) server_negotiation.cc:299] Negotiation successful
0928 04:57:58.918690 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 04:57:58.919389 18157 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46003
I20260928 04:57:58.919525 18157 heartbeater.cc:464] Registering TS with master...
I20260928 04:57:58.919773 18157 heartbeater.cc:511] Master 127.0.98.62:46003 requested a full tablet report, sending...
I20260928 04:57:58.920497 17969 ts_manager.cc:194] Registered new tserver with Master: 8c0d3bcab0814e469f3bce736b07c4e7 (127.0.98.1:42089)
I20260928 04:57:58.921590 17969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:37515
W20260928 04:57:59.009152 18160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:57:59.009413 18160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:57:59.009440 18160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:57:59.010856 18160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:57:59.010917 18160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:57:59.010947 18160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:57:59.012599 18160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:57:59.012853 18160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:57:59.013223 18160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:57:59.019816 18166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:59.019795 18168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:59.019795 18165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:59.019845 18160 server_base.cc:1034] running on GCE node
I20260928 04:57:59.020337 18160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:57:59.020586 18160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:57:59.021749 18160 hybrid_clock.cc:648] HybridClock initialized: now 1790571479021738 us; error 31 us; skew 500 ppm
I20260928 04:57:59.023241 18160 webserver.cc:460] Webserver started at http://127.0.98.2:33943/ using document root <none> and password file <none>
I20260928 04:57:59.023510 18160 fs_manager.cc:359] Metadata directory not provided
I20260928 04:57:59.023613 18160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:57:59.023909 18160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:57:59.025233 18160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "56f24facd3ec4a22b40e73f68d6a0b6f"
format_stamp: "Formatted at 2026-09-28 04:57:59 on dist-test-slave-m39l"
server_key: "67574dc85742dbb1d2ef2ae8ce72c126"
server_key_iv: "c5af5412bad25b6b8080c6b406cd0e6b"
server_key_version: "encryptionkey@0"
I20260928 04:57:59.025620 18160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "56f24facd3ec4a22b40e73f68d6a0b6f"
format_stamp: "Formatted at 2026-09-28 04:57:59 on dist-test-slave-m39l"
server_key: "67574dc85742dbb1d2ef2ae8ce72c126"
server_key_iv: "c5af5412bad25b6b8080c6b406cd0e6b"
server_key_version: "encryptionkey@0"
I20260928 04:57:59.027267 18160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:57:59.028306 18174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:59.028559 18160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:57:59.028664 18160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "56f24facd3ec4a22b40e73f68d6a0b6f"
format_stamp: "Formatted at 2026-09-28 04:57:59 on dist-test-slave-m39l"
server_key: "67574dc85742dbb1d2ef2ae8ce72c126"
server_key_iv: "c5af5412bad25b6b8080c6b406cd0e6b"
server_key_version: "encryptionkey@0"
I20260928 04:57:59.028733 18160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:57:59.062235 18160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:57:59.062696 18160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:57:59.062832 18160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:57:59.063053 18160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:57:59.063432 18160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:57:59.063476 18160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:59.063503 18160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:57:59.063552 18160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:59.070591 18160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39539
I20260928 04:57:59.070785 18287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39539 every 8 connection(s)
I20260928 04:57:59.071076 18160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:57:59.074625 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 18160
I20260928 04:57:59.074779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:57:59.075038 392 external_mini_cluster.cc:1428] Setting key 4d7d67e27d68f19bf8c500c2e458eb0c
I20260928 04:57:59.076175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46003
--builtin_ntp_servers=127.0.98.20:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 04:57:59.077431 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:59.071870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40247 (local address 127.0.98.62:46003)
0928 04:57:59.071915 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:59.071918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:59.072263 (+ 345us) server_negotiation.cc:407] Connection header received
0928 04:57:59.072792 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:59.072793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:59.072806 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:59.072862 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:57:59.073599 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.074124 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.075404 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.075770 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.075866 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:59.077102 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:57:59.077103 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:57:59.077104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:57:59.077110 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:57:59.077130 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:57:59.077135 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:57:59.077204 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:59.077282 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 04:57:59.077313 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":19}
I20260928 04:57:59.077708 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:59.071834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46003 (local address 127.0.98.2:40247)
0928 04:57:59.072070 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:59.072108 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:59.072673 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:59.072956 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:59.072960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:59.072968 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:57:59.073496 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:59.073500 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.074248 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.074249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:59.075308 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:59.075313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.075957 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.075959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:59.076163 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:59.076909 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:57:59.076910 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:57:59.076955 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:57:59.076988 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:57:59.077204 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:57:59.077207 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:57:59.077210 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:57:59.077267 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 04:57:59.077408 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":114,"threads_started":1}
I20260928 04:57:59.078312 18288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46003
I20260928 04:57:59.078452 18288 heartbeater.cc:464] Registering TS with master...
I20260928 04:57:59.078706 18288 heartbeater.cc:511] Master 127.0.98.62:46003 requested a full tablet report, sending...
I20260928 04:57:59.079130 17969 ts_manager.cc:194] Registered new tserver with Master: 56f24facd3ec4a22b40e73f68d6a0b6f (127.0.98.2:39539)
I20260928 04:57:59.079531 17969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40247
W20260928 04:57:59.164980 18291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:57:59.165225 18291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:57:59.165249 18291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:57:59.166565 18291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:57:59.166623 18291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:57:59.166653 18291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:57:59.168836 18291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:57:59.169211 18291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:57:59.169637 18291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:57:59.176509 18291 server_base.cc:1034] running on GCE node
W20260928 04:57:59.176424 18297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:59.176448 18299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:57:59.176414 18296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:57:59.177075 18291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:57:59.177419 18291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:57:59.178606 18291 hybrid_clock.cc:648] HybridClock initialized: now 1790571479178580 us; error 48 us; skew 500 ppm
I20260928 04:57:59.180155 18291 webserver.cc:460] Webserver started at http://127.0.98.3:37617/ using document root <none> and password file <none>
I20260928 04:57:59.180451 18291 fs_manager.cc:359] Metadata directory not provided
I20260928 04:57:59.180538 18291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:57:59.180709 18291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:57:59.182051 18291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "6cdb184c400341c0bbeac01621092aa5"
format_stamp: "Formatted at 2026-09-28 04:57:59 on dist-test-slave-m39l"
server_key: "1308bd4e055a88242d0932a6f22411d4"
server_key_iv: "5ab6a20a779d424478df2daeb810dc65"
server_key_version: "encryptionkey@0"
I20260928 04:57:59.182457 18291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6cdb184c400341c0bbeac01621092aa5"
format_stamp: "Formatted at 2026-09-28 04:57:59 on dist-test-slave-m39l"
server_key: "1308bd4e055a88242d0932a6f22411d4"
server_key_iv: "5ab6a20a779d424478df2daeb810dc65"
server_key_version: "encryptionkey@0"
I20260928 04:57:59.184118 18291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:57:59.185108 18305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:59.185377 18291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:57:59.185535 18291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "6cdb184c400341c0bbeac01621092aa5"
format_stamp: "Formatted at 2026-09-28 04:57:59 on dist-test-slave-m39l"
server_key: "1308bd4e055a88242d0932a6f22411d4"
server_key_iv: "5ab6a20a779d424478df2daeb810dc65"
server_key_version: "encryptionkey@0"
I20260928 04:57:59.185629 18291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:57:59.200420 18291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:57:59.200819 18291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:57:59.200991 18291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:57:59.201242 18291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:57:59.201598 18291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:57:59.201658 18291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:59.201704 18291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:57:59.201728 18291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:57:59.207756 18291 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37535
I20260928 04:57:59.207804 18418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37535 every 8 connection(s)
I20260928 04:57:59.208186 18291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:57:59.212515 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 18291
I20260928 04:57:59.212677 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:57:59.212880 392 external_mini_cluster.cc:1428] Setting key 392297642f70a20e0723188cd80e3bfe
I20260928 04:57:59.214416 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:59.209073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34593 (local address 127.0.98.62:46003)
0928 04:57:59.209170 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:59.209173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:59.209388 (+ 215us) server_negotiation.cc:407] Connection header received
0928 04:57:59.209997 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:59.209998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:59.210079 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:59.210166 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:57:59.210984 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.211538 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.212610 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.212878 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.212976 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:59.214117 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:57:59.214118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:57:59.214118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:57:59.214123 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:57:59.214141 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:57:59.214147 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:57:59.214218 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:59.214303 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 04:57:59.214333 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:57:59.214691 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:59.209031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46003 (local address 127.0.98.3:34593)
0928 04:57:59.209260 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:59.209296 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:59.209861 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:59.210255 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:59.210258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:59.210268 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:57:59.210878 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:59.210884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.211640 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.211641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:59.212510 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:59.212512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.212978 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.212980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:59.213162 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:59.213926 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:57:59.213926 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:57:59.213967 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:57:59.214002 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:57:59.214218 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:57:59.214225 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:57:59.214229 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:57:59.214299 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:57:59.214414 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":89,"threads_started":1}
I20260928 04:57:59.215159 18419 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46003
I20260928 04:57:59.215306 18419 heartbeater.cc:464] Registering TS with master...
I20260928 04:57:59.215516 18419 heartbeater.cc:511] Master 127.0.98.62:46003 requested a full tablet report, sending...
I20260928 04:57:59.215936 17969 ts_manager.cc:194] Registered new tserver with Master: 6cdb184c400341c0bbeac01621092aa5 (127.0.98.3:37535)
I20260928 04:57:59.216346 17969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34593
I20260928 04:57:59.225931 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:57:59.231087 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:59.227690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40514 (local address 127.0.98.62:46003)
0928 04:57:59.227747 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:59.227750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:59.227787 (+ 37us) server_negotiation.cc:407] Connection header received
0928 04:57:59.227870 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:59.227870 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:59.227879 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:59.227934 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:57:59.228223 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.228734 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.229539 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.229730 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.229797 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:59.230770 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:57:59.230771 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:57:59.230771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:57:59.230777 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:57:59.230790 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:57:59.230794 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:57:59.230899 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:59.230993 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 04:57:59.231019 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:57:59.233278 17969 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:57:59.235821 17969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:57:59.243162 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:59.238796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58918 (local address 127.0.98.1:42089)
0928 04:57:59.239455 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:59.239460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:59.239470 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:57:59.239515 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:59.239517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:59.239535 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:59.239618 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:57:59.239729 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.240816 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.242129 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.242734 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.242840 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:59.242954 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:59.243018 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 04:57:59.243057 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":104,"threads_started":1}
I20260928 04:57:59.243291 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:59.239010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45680 (local address 127.0.98.2:39539)
0928 04:57:59.239302 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:59.239306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:59.239316 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:57:59.239385 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:59.239387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:59.239403 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:59.239499 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:57:59.239655 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.240421 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.242021 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.242782 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.243021 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:59.243132 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:59.243159 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 04:57:59.243202 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":184,"threads_started":1}
I20260928 04:57:59.243309 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:59.238518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39539 (local address 127.0.0.1:45680)
0928 04:57:59.239008 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:59.239021 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:59.239084 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:59.239497 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:59.239498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:59.239499 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:57:59.239563 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:59.239565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.240551 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.240553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:59.241905 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:59.241908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.242867 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.242868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:59.242915 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:59.242918 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:57:59.242970 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 04:57:59.243223 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":240,"thread_start_us":111,"threads_started":1}
I20260928 04:57:59.243185 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:59.238459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37535 (local address 127.0.0.1:51496)
0928 04:57:59.238643 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:59.238689 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:59.238779 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:59.239258 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:59.239261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:59.239264 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:57:59.239349 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:59.239351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.240785 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.240787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:59.241798 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:59.241801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.242803 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.242804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:59.242897 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:59.242915 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 04:57:59.242970 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 04:57:59.243053 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":1,"thread_start_us":111,"threads_started":1}
I20260928 04:57:59.243183 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:59.238466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42089 (local address 127.0.0.1:58918)
0928 04:57:59.238880 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:59.238890 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:59.239105 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:59.239614 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:59.239615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:59.239617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:57:59.239653 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:59.239654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.240912 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.240913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:59.242022 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:59.242024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.242842 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.242843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:59.242908 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:59.242918 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 04:57:59.243000 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 04:57:59.243057 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":3,"thread_start_us":100,"threads_started":1}
I20260928 04:57:59.243686 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:59.238804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51496 (local address 127.0.98.3:37535)
0928 04:57:59.239039 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:59.239044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:59.239057 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:57:59.239104 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:59.239107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:59.239125 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:59.239249 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:57:59.239485 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.240670 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.241891 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.242690 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.243402 (+ 712us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:59.243501 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:59.243529 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 04:57:59.243588 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":99,"threads_started":1}
I20260928 04:57:59.244474 18353 tablet_service.cc:1467] Processing CreateTablet for tablet ad38077340524fe29cbbd5e6bbaf340e (DEFAULT_TABLE table=test-table [id=39376ec06f4e4676b2b84ee3e7b5c87b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:57:59.244654 18222 tablet_service.cc:1467] Processing CreateTablet for tablet ad38077340524fe29cbbd5e6bbaf340e (DEFAULT_TABLE table=test-table [id=39376ec06f4e4676b2b84ee3e7b5c87b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:57:59.244697 18091 tablet_service.cc:1467] Processing CreateTablet for tablet ad38077340524fe29cbbd5e6bbaf340e (DEFAULT_TABLE table=test-table [id=39376ec06f4e4676b2b84ee3e7b5c87b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:57:59.244946 18353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad38077340524fe29cbbd5e6bbaf340e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:57:59.245018 18222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad38077340524fe29cbbd5e6bbaf340e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:57:59.245018 18091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad38077340524fe29cbbd5e6bbaf340e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:57:59.248248 18433 tablet_bootstrap.cc:492] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5: Bootstrap starting.
I20260928 04:57:59.248514 18434 tablet_bootstrap.cc:492] T ad38077340524fe29cbbd5e6bbaf340e P 8c0d3bcab0814e469f3bce736b07c4e7: Bootstrap starting.
I20260928 04:57:59.248692 18435 tablet_bootstrap.cc:492] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f: Bootstrap starting.
I20260928 04:57:59.249119 18433 tablet_bootstrap.cc:654] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5: Neither blocks nor log segments found. Creating new log.
I20260928 04:57:59.249257 18434 tablet_bootstrap.cc:654] T ad38077340524fe29cbbd5e6bbaf340e P 8c0d3bcab0814e469f3bce736b07c4e7: Neither blocks nor log segments found. Creating new log.
I20260928 04:57:59.249279 18435 tablet_bootstrap.cc:654] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f: Neither blocks nor log segments found. Creating new log.
I20260928 04:57:59.249486 18433 log.cc:824] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5: Log is configured to *not* fsync() on all Append() calls
I20260928 04:57:59.249598 18434 log.cc:824] T ad38077340524fe29cbbd5e6bbaf340e P 8c0d3bcab0814e469f3bce736b07c4e7: Log is configured to *not* fsync() on all Append() calls
I20260928 04:57:59.249600 18435 log.cc:824] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f: Log is configured to *not* fsync() on all Append() calls
I20260928 04:57:59.250424 18433 tablet_bootstrap.cc:492] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5: No bootstrap required, opened a new log
I20260928 04:57:59.250540 18433 ts_tablet_manager.cc:1397] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:57:59.250741 18434 tablet_bootstrap.cc:492] T ad38077340524fe29cbbd5e6bbaf340e P 8c0d3bcab0814e469f3bce736b07c4e7: No bootstrap required, opened a new log
I20260928 04:57:59.250824 18434 ts_tablet_manager.cc:1397] T ad38077340524fe29cbbd5e6bbaf340e P 8c0d3bcab0814e469f3bce736b07c4e7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:57:59.250852 18435 tablet_bootstrap.cc:492] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f: No bootstrap required, opened a new log
I20260928 04:57:59.250923 18435 ts_tablet_manager.cc:1397] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:57:59.253002 18433 raft_consensus.cc:348] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0d3bcab0814e469f3bce736b07c4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42089 } } peers { permanent_uuid: "56f24facd3ec4a22b40e73f68d6a0b6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39539 } } peers { permanent_uuid: "6cdb184c400341c0bbeac01621092aa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } }
I20260928 04:57:59.253232 18433 raft_consensus.cc:374] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:57:59.253284 18433 raft_consensus.cc:729] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cdb184c400341c0bbeac01621092aa5, State: Initialized, Role: FOLLOWER
I20260928 04:57:59.253408 18433 consensus_queue.cc:260] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0d3bcab0814e469f3bce736b07c4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42089 } } peers { permanent_uuid: "56f24facd3ec4a22b40e73f68d6a0b6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39539 } } peers { permanent_uuid: "6cdb184c400341c0bbeac01621092aa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } }
I20260928 04:57:59.253700 18433 ts_tablet_manager.cc:1428] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:57:59.253849 18419 heartbeater.cc:503] Master 127.0.98.62:46003 was elected leader, sending a full tablet report...
I20260928 04:57:59.253916 18435 raft_consensus.cc:348] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0d3bcab0814e469f3bce736b07c4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42089 } } peers { permanent_uuid: "56f24facd3ec4a22b40e73f68d6a0b6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39539 } } peers { permanent_uuid: "6cdb184c400341c0bbeac01621092aa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } }
I20260928 04:57:59.254105 18435 raft_consensus.cc:374] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:57:59.254160 18435 raft_consensus.cc:729] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56f24facd3ec4a22b40e73f68d6a0b6f, State: Initialized, Role: FOLLOWER
I20260928 04:57:59.254334 18435 consensus_queue.cc:260] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0d3bcab0814e469f3bce736b07c4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42089 } } peers { permanent_uuid: "56f24facd3ec4a22b40e73f68d6a0b6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39539 } } peers { permanent_uuid: "6cdb184c400341c0bbeac01621092aa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } }
I20260928 04:57:59.254645 18435 ts_tablet_manager.cc:1428] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 04:57:59.254694 18288 heartbeater.cc:503] Master 127.0.98.62:46003 was elected leader, sending a full tablet report...
I20260928 04:57:59.253791 18434 raft_consensus.cc:348] T ad38077340524fe29cbbd5e6bbaf340e P 8c0d3bcab0814e469f3bce736b07c4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0d3bcab0814e469f3bce736b07c4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42089 } } peers { permanent_uuid: "56f24facd3ec4a22b40e73f68d6a0b6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39539 } } peers { permanent_uuid: "6cdb184c400341c0bbeac01621092aa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } }
I20260928 04:57:59.254995 18434 raft_consensus.cc:374] T ad38077340524fe29cbbd5e6bbaf340e P 8c0d3bcab0814e469f3bce736b07c4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:57:59.255056 18434 raft_consensus.cc:729] T ad38077340524fe29cbbd5e6bbaf340e P 8c0d3bcab0814e469f3bce736b07c4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c0d3bcab0814e469f3bce736b07c4e7, State: Initialized, Role: FOLLOWER
I20260928 04:57:59.255249 18434 consensus_queue.cc:260] T ad38077340524fe29cbbd5e6bbaf340e P 8c0d3bcab0814e469f3bce736b07c4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0d3bcab0814e469f3bce736b07c4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42089 } } peers { permanent_uuid: "56f24facd3ec4a22b40e73f68d6a0b6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39539 } } peers { permanent_uuid: "6cdb184c400341c0bbeac01621092aa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } }
I20260928 04:57:59.255600 18157 heartbeater.cc:503] Master 127.0.98.62:46003 was elected leader, sending a full tablet report...
I20260928 04:57:59.255589 18434 ts_tablet_manager.cc:1428] T ad38077340524fe29cbbd5e6bbaf340e P 8c0d3bcab0814e469f3bce736b07c4e7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260928 04:57:59.321718 18289 tablet.cc:2369] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:57:59.411790 18158 tablet.cc:2369] T ad38077340524fe29cbbd5e6bbaf340e P 8c0d3bcab0814e469f3bce736b07c4e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:57:59.418473 18439 raft_consensus.cc:482] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:57:59.418713 18439 raft_consensus.cc:504] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0d3bcab0814e469f3bce736b07c4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42089 } } peers { permanent_uuid: "56f24facd3ec4a22b40e73f68d6a0b6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39539 } } peers { permanent_uuid: "6cdb184c400341c0bbeac01621092aa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } }
I20260928 04:57:59.419229 18439 leader_election.cc:288] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c0d3bcab0814e469f3bce736b07c4e7 (127.0.98.1:42089), 56f24facd3ec4a22b40e73f68d6a0b6f (127.0.98.2:39539)
I20260928 04:57:59.424217 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:59.419441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42089 (local address 127.0.98.3:45897)
0928 04:57:59.419556 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:59.419568 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:59.419620 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:59.419832 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:59.419833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:59.419834 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:57:59.419934 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:59.419937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.420944 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.420945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:59.422459 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:59.422465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.423823 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.423825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:59.423950 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:59.423958 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 04:57:59.424036 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 04:57:59.424100 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":177}
I20260928 04:57:59.424314 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:59.419535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45897 (local address 127.0.98.1:42089)
0928 04:57:59.419591 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:59.419594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:59.419606 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:57:59.419720 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:59.419721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:59.419754 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:59.419810 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:57:59.420085 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.420842 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.422908 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.423611 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.423750 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:59.423763 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:59.424171 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 04:57:59.424228 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 04:57:59.424314 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:59.419571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48079 (local address 127.0.98.2:39539)
0928 04:57:59.419627 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:59.419630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:59.419781 (+ 151us) server_negotiation.cc:407] Connection header received
0928 04:57:59.419809 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:59.419809 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:59.419822 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:59.419872 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:57:59.420083 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.420839 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.422908 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.423656 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.423750 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:59.423784 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:59.424174 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 04:57:59.424231 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:57:59.424222 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:59.419490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39539 (local address 127.0.98.3:48079)
0928 04:57:59.419673 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:59.419687 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:57:59.419743 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:57:59.419888 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:57:59.419889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:57:59.419890 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:57:59.419946 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:59.419947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.420944 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.420945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:59.422459 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:57:59.422465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.423823 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.423825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:57:59.423950 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:59.423958 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 04:57:59.424036 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 04:57:59.424097 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":86,"thread_start_us":122,"threads_started":1}
I20260928 04:57:59.424898 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad38077340524fe29cbbd5e6bbaf340e" candidate_uuid: "6cdb184c400341c0bbeac01621092aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c0d3bcab0814e469f3bce736b07c4e7" is_pre_election: true
I20260928 04:57:59.425089 18242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad38077340524fe29cbbd5e6bbaf340e" candidate_uuid: "6cdb184c400341c0bbeac01621092aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56f24facd3ec4a22b40e73f68d6a0b6f" is_pre_election: true
I20260928 04:57:59.425169 18111 raft_consensus.cc:2454] T ad38077340524fe29cbbd5e6bbaf340e P 8c0d3bcab0814e469f3bce736b07c4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cdb184c400341c0bbeac01621092aa5 in term 0.
I20260928 04:57:59.425227 18242 raft_consensus.cc:2454] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cdb184c400341c0bbeac01621092aa5 in term 0.
I20260928 04:57:59.425484 18309 leader_election.cc:302] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cdb184c400341c0bbeac01621092aa5, 8c0d3bcab0814e469f3bce736b07c4e7; no voters:
I20260928 04:57:59.425664 18439 raft_consensus.cc:2781] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:57:59.425735 18439 raft_consensus.cc:482] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:57:59.425767 18439 raft_consensus.cc:3037] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:57:59.426767 18439 raft_consensus.cc:504] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0d3bcab0814e469f3bce736b07c4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42089 } } peers { permanent_uuid: "56f24facd3ec4a22b40e73f68d6a0b6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39539 } } peers { permanent_uuid: "6cdb184c400341c0bbeac01621092aa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } }
I20260928 04:57:59.426965 18439 leader_election.cc:288] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [CANDIDATE]: Term 1 election: Requested vote from peers 8c0d3bcab0814e469f3bce736b07c4e7 (127.0.98.1:42089), 56f24facd3ec4a22b40e73f68d6a0b6f (127.0.98.2:39539)
I20260928 04:57:59.427168 18242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad38077340524fe29cbbd5e6bbaf340e" candidate_uuid: "6cdb184c400341c0bbeac01621092aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56f24facd3ec4a22b40e73f68d6a0b6f"
I20260928 04:57:59.427155 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad38077340524fe29cbbd5e6bbaf340e" candidate_uuid: "6cdb184c400341c0bbeac01621092aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c0d3bcab0814e469f3bce736b07c4e7"
I20260928 04:57:59.427273 18242 raft_consensus.cc:3037] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:57:59.427281 18111 raft_consensus.cc:3037] T ad38077340524fe29cbbd5e6bbaf340e P 8c0d3bcab0814e469f3bce736b07c4e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:57:59.428323 18111 raft_consensus.cc:2454] T ad38077340524fe29cbbd5e6bbaf340e P 8c0d3bcab0814e469f3bce736b07c4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cdb184c400341c0bbeac01621092aa5 in term 1.
I20260928 04:57:59.428323 18242 raft_consensus.cc:2454] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cdb184c400341c0bbeac01621092aa5 in term 1.
I20260928 04:57:59.428613 18309 leader_election.cc:302] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cdb184c400341c0bbeac01621092aa5, 8c0d3bcab0814e469f3bce736b07c4e7; no voters:
I20260928 04:57:59.428762 18439 raft_consensus.cc:2781] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:57:59.428932 18439 raft_consensus.cc:686] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [term 1 LEADER]: Becoming Leader. State: Replica: 6cdb184c400341c0bbeac01621092aa5, State: Running, Role: LEADER
I20260928 04:57:59.429047 18439 consensus_queue.cc:237] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0d3bcab0814e469f3bce736b07c4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42089 } } peers { permanent_uuid: "56f24facd3ec4a22b40e73f68d6a0b6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39539 } } peers { permanent_uuid: "6cdb184c400341c0bbeac01621092aa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } }
I20260928 04:57:59.430025 17968 catalog_manager.cc:5495] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cdb184c400341c0bbeac01621092aa5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6cdb184c400341c0bbeac01621092aa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0d3bcab0814e469f3bce736b07c4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56f24facd3ec4a22b40e73f68d6a0b6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cdb184c400341c0bbeac01621092aa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } health_report { overall_health: HEALTHY } } }
W20260928 04:57:59.458837 18420 tablet.cc:2369] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:57:59.477617 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:59.474719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.0.98.3:37535)
0928 04:57:59.474783 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 04:57:59.474785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:57:59.474905 (+ 120us) server_negotiation.cc:407] Connection header received
0928 04:57:59.474961 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:57:59.474962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:57:59.474995 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:57:59.475072 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:57:59.475357 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.475942 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.476752 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:57:59.476957 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:57:59.477039 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:57:59.477235 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:57:59.477347 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:57:59.477448 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 04:57:59.477468 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 04:57:59.477489 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:57:59.479629 18242 raft_consensus.cc:1261] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f [term 1 FOLLOWER]: Refusing update from remote peer 6cdb184c400341c0bbeac01621092aa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:57:59.479629 18111 raft_consensus.cc:1261] T ad38077340524fe29cbbd5e6bbaf340e P 8c0d3bcab0814e469f3bce736b07c4e7 [term 1 FOLLOWER]: Refusing update from remote peer 6cdb184c400341c0bbeac01621092aa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:57:59.480415 18439 consensus_queue.cc:1035] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c0d3bcab0814e469f3bce736b07c4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:57:59.480633 18443 consensus_queue.cc:1035] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56f24facd3ec4a22b40e73f68d6a0b6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:57:59.483723 18446 mvcc.cc:205] Tried to move back new op lower bound from 7334180779945447424 to 7334180779742285824. Current Snapshot: MvccSnapshot[applied={T|T < 7334180779945447424}]
I20260928 04:57:59.484458 18447 mvcc.cc:205] Tried to move back new op lower bound from 7334180779945447424 to 7334180779742285824. Current Snapshot: MvccSnapshot[applied={T|T < 7334180779945447424}]
I20260928 04:57:59.486140 18448 mvcc.cc:205] Tried to move back new op lower bound from 7334180779945447424 to 7334180779742285824. Current Snapshot: MvccSnapshot[applied={T|T < 7334180779945447424 or (T in {7334180779945447424})}]
I20260928 04:57:59.486114 17969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:57:59.486234 17969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:57:59.487118 17969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7f347c6264e45b48fcc4236bde5eaba: Sending DeleteTablet for 3 replicas of tablet ad38077340524fe29cbbd5e6bbaf340e
I20260928 04:57:59.487550 18222 tablet_service.cc:1514] Processing DeleteTablet for tablet ad38077340524fe29cbbd5e6bbaf340e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:57:59 UTC) from {username='slave'} at 127.0.0.1:45680
I20260928 04:57:59.487620 18091 tablet_service.cc:1514] Processing DeleteTablet for tablet ad38077340524fe29cbbd5e6bbaf340e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:57:59 UTC) from {username='slave'} at 127.0.0.1:58918
I20260928 04:57:59.487887 18456 tablet_replica.cc:330] T ad38077340524fe29cbbd5e6bbaf340e P 8c0d3bcab0814e469f3bce736b07c4e7: stopping tablet replica
I20260928 04:57:59.487898 18455 tablet_replica.cc:330] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f: stopping tablet replica
I20260928 04:57:59.488010 18455 raft_consensus.cc:2229] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:57:59.488030 18353 tablet_service.cc:1514] Processing DeleteTablet for tablet ad38077340524fe29cbbd5e6bbaf340e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:57:59 UTC) from {username='slave'} at 127.0.0.1:51496
I20260928 04:57:59.488212 18455 raft_consensus.cc:2258] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:57:59.488469 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 18029
I20260928 04:57:59.489125 18457 tablet_replica.cc:330] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5: stopping tablet replica
I20260928 04:57:59.489296 18457 raft_consensus.cc:2229] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:57:59.489478 18457 raft_consensus.cc:2258] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:57:59.489708 18455 ts_tablet_manager.cc:1905] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:57:59.491856 18455 ts_tablet_manager.cc:1918] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:57:59.491971 18455 log.cc:1196] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/ad38077340524fe29cbbd5e6bbaf340e
I20260928 04:57:59.492306 18455 ts_tablet_manager.cc:1939] T ad38077340524fe29cbbd5e6bbaf340e P 56f24facd3ec4a22b40e73f68d6a0b6f: Deleting consensus metadata
I20260928 04:57:59.492832 17954 catalog_manager.cc:4841] TS 56f24facd3ec4a22b40e73f68d6a0b6f (127.0.98.2:39539): tablet ad38077340524fe29cbbd5e6bbaf340e (table test-table [id=39376ec06f4e4676b2b84ee3e7b5c87b]) successfully deleted
W20260928 04:57:59.494648 17956 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42089 (error 108)
I20260928 04:57:59.495379 18457 ts_tablet_manager.cc:1905] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:57:59.495571 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:57:59.495303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42089 (local address 127.0.0.1:58928)
0928 04:57:59.495376 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 04:57:59.495527 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 04:57:59.495774 17956 catalog_manager.cc:4584] TS 8c0d3bcab0814e469f3bce736b07c4e7 (127.0.98.1:42089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad38077340524fe29cbbd5e6bbaf340e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42089: connect: Connection refused (error 111)
I20260928 04:57:59.497367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 18160
I20260928 04:57:59.497412 18457 ts_tablet_manager.cc:1918] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:57:59.498445 18457 log.cc:1196] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/ad38077340524fe29cbbd5e6bbaf340e
I20260928 04:57:59.498832 18457 ts_tablet_manager.cc:1939] T ad38077340524fe29cbbd5e6bbaf340e P 6cdb184c400341c0bbeac01621092aa5: Deleting consensus metadata
I20260928 04:57:59.499382 17955 catalog_manager.cc:4841] TS 6cdb184c400341c0bbeac01621092aa5 (127.0.98.3:37535): tablet ad38077340524fe29cbbd5e6bbaf340e (table test-table [id=39376ec06f4e4676b2b84ee3e7b5c87b]) successfully deleted
I20260928 04:57:59.504454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 18291
I20260928 04:57:59.511569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 17938
2026-09-28T04:57:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (920 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:57:59 dist-test-slave-m39l krb5kdc[18461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:57:59 dist-test-slave-m39l krb5kdc[18461](info): set up 2 sockets
Sep 28 04:57:59 dist-test-slave-m39l krb5kdc[18461](info): commencing operation
krb5kdc: starting...
W20260928 04:58:00.654393 392 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:00 dist-test-slave-m39l krb5kdc[18461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:58:00.787243 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45521
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:00.871519 18474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:00.871865 18474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:00.871898 18474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:00.873301 18474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:00.873364 18474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:00.873380 18474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:00.873394 18474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:00.873405 18474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:00.875089 18474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:00.875396 18474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:00.875651 18474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:00.881222 18482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:00.881259 18480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:00.881237 18479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:00.881772 18474 server_base.cc:1034] running on GCE node
I20260928 04:58:00.882078 18474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:00.882376 18474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:00.883553 18474 hybrid_clock.cc:648] HybridClock initialized: now 1790571480883545 us; error 44 us; skew 500 ppm
Sep 28 04:58:00 dist-test-slave-m39l krb5kdc[18461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:00.886319 18474 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:58:00.886965 18474 webserver.cc:460] Webserver started at http://127.0.98.62:35813/ using document root <none> and password file <none>
I20260928 04:58:00.887224 18474 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:00.887300 18474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:00.887477 18474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:00.888756 18474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "b29934b42b3242fc91ccba9f41e2d7fe"
format_stamp: "Formatted at 2026-09-28 04:58:00 on dist-test-slave-m39l"
server_key: "4eb46410ae6ceb3d1238fdc785e700cd"
server_key_iv: "c4af81390419772c714f00900dede9f3"
server_key_version: "encryptionkey@0"
I20260928 04:58:00.889457 18474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "b29934b42b3242fc91ccba9f41e2d7fe"
format_stamp: "Formatted at 2026-09-28 04:58:00 on dist-test-slave-m39l"
server_key: "4eb46410ae6ceb3d1238fdc785e700cd"
server_key_iv: "c4af81390419772c714f00900dede9f3"
server_key_version: "encryptionkey@0"
I20260928 04:58:00.891194 18474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260928 04:58:00.892194 18489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:00.892378 18474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:00.892489 18474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "b29934b42b3242fc91ccba9f41e2d7fe"
format_stamp: "Formatted at 2026-09-28 04:58:00 on dist-test-slave-m39l"
server_key: "4eb46410ae6ceb3d1238fdc785e700cd"
server_key_iv: "c4af81390419772c714f00900dede9f3"
server_key_version: "encryptionkey@0"
I20260928 04:58:00.892580 18474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:00.909049 18474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:00.911895 18474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:00.912189 18474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:00.916328 18474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45521
I20260928 04:58:00.916369 18541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45521 every 8 connection(s)
I20260928 04:58:00.916774 18474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:00.917701 18542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:00.920176 18542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe: Bootstrap starting.
I20260928 04:58:00.920804 18542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:00.921108 18542 log.cc:824] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:00.921845 18542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe: No bootstrap required, opened a new log
I20260928 04:58:00.922726 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 18474
I20260928 04:58:00.922845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:00.923090 392 external_mini_cluster.cc:1428] Setting key 649e4e3a8446c1173812d7edafcd2ae7
I20260928 04:58:00.924084 18542 raft_consensus.cc:348] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29934b42b3242fc91ccba9f41e2d7fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45521 } }
I20260928 04:58:00.924278 18542 raft_consensus.cc:374] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:00.924325 18542 raft_consensus.cc:729] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b29934b42b3242fc91ccba9f41e2d7fe, State: Initialized, Role: FOLLOWER
I20260928 04:58:00.924469 18542 consensus_queue.cc:260] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29934b42b3242fc91ccba9f41e2d7fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45521 } }
I20260928 04:58:00.924566 18542 raft_consensus.cc:388] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:00.924602 18542 raft_consensus.cc:482] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:00.924683 18542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:00.925326 18542 raft_consensus.cc:504] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29934b42b3242fc91ccba9f41e2d7fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45521 } }
I20260928 04:58:00.925426 18542 leader_election.cc:302] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b29934b42b3242fc91ccba9f41e2d7fe; no voters:
I20260928 04:58:00.925671 18542 leader_election.cc:288] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:00.925690 18547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:00.925812 18547 raft_consensus.cc:686] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe [term 1 LEADER]: Becoming Leader. State: Replica: b29934b42b3242fc91ccba9f41e2d7fe, State: Running, Role: LEADER
I20260928 04:58:00.925899 18547 consensus_queue.cc:237] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29934b42b3242fc91ccba9f41e2d7fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45521 } }
I20260928 04:58:00.925973 18542 sys_catalog.cc:564] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:00.926275 18548 sys_catalog.cc:455] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b29934b42b3242fc91ccba9f41e2d7fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29934b42b3242fc91ccba9f41e2d7fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45521 } } }
I20260928 04:58:00.926365 18548 sys_catalog.cc:458] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:00.926630 18552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:00.926307 18549 sys_catalog.cc:455] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader b29934b42b3242fc91ccba9f41e2d7fe. Latest consensus state: current_term: 1 leader_uuid: "b29934b42b3242fc91ccba9f41e2d7fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29934b42b3242fc91ccba9f41e2d7fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45521 } } }
I20260928 04:58:00.926923 18549 sys_catalog.cc:458] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:00.927218 18552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:00.929003 18552 catalog_manager.cc:1347] Generated new cluster ID: f3bc2bb063354dd29a97315a202f7ab3
I20260928 04:58:00.929057 18552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:58:00 dist-test-slave-m39l krb5kdc[18461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:58:00.936190 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:00.923728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53800 (local address 127.0.98.62:45521)
0928 04:58:00.923962 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:00.923973 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:00.924014 (+ 41us) server_negotiation.cc:407] Connection header received
0928 04:58:00.924142 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:00.924150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:00.924220 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:00.924385 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:00.925614 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:00.927155 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:00.928311 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:00.928920 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:00.929396 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:00.931997 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:00.932003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:00.932003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:00.932017 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:00.934254 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:00.934728 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:00.934734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:00.934734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:00.934905 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:00.935126 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:00.935128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:00.935128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:00.935532 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:00.935708 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:00.935790 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 04:58:00.935946 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":107,"threads_started":1}
I20260928 04:58:00.947429 18552 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:00.948192 18552 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:00.953837 18552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe: Generated new TSK 0
I20260928 04:58:00.954066 18552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:58:00.987294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45521
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:01.076838 18568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:01.077133 18568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:01.077167 18568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:01.078596 18568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:01.078655 18568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:01.078686 18568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:58:01.080487 18568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:01.080741 18568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:01.080950 18568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:01.086845 18576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:01.086972 18568 server_base.cc:1034] running on GCE node
W20260928 04:58:01.086845 18574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:01.086870 18573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:01.087546 18568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:01.087823 18568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:01.088990 18568 hybrid_clock.cc:648] HybridClock initialized: now 1790571481088959 us; error 51 us; skew 500 ppm
Sep 28 04:58:01 dist-test-slave-m39l krb5kdc[18461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:01.092262 18568 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:58:01.092779 18568 webserver.cc:460] Webserver started at http://127.0.98.1:37661/ using document root <none> and password file <none>
I20260928 04:58:01.093029 18568 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:01.093099 18568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:01.093261 18568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:01.094408 18568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "12ad1286a81948489805dcde77cee765"
format_stamp: "Formatted at 2026-09-28 04:58:01 on dist-test-slave-m39l"
server_key: "bae8a61ec4422896be5aa37ca87c14e1"
server_key_iv: "ac86bb1539644a76a0fe9ae3832deec5"
server_key_version: "encryptionkey@0"
I20260928 04:58:01.094857 18568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "12ad1286a81948489805dcde77cee765"
format_stamp: "Formatted at 2026-09-28 04:58:01 on dist-test-slave-m39l"
server_key: "bae8a61ec4422896be5aa37ca87c14e1"
server_key_iv: "ac86bb1539644a76a0fe9ae3832deec5"
server_key_version: "encryptionkey@0"
I20260928 04:58:01.096432 18568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:58:01.097429 18583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:01.097734 18568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:01.097842 18568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "12ad1286a81948489805dcde77cee765"
format_stamp: "Formatted at 2026-09-28 04:58:01 on dist-test-slave-m39l"
server_key: "bae8a61ec4422896be5aa37ca87c14e1"
server_key_iv: "ac86bb1539644a76a0fe9ae3832deec5"
server_key_version: "encryptionkey@0"
I20260928 04:58:01.097918 18568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:01.114372 18568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:01.117004 18568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:01.117228 18568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:01.117509 18568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:58:01.117864 18568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:58:01.117900 18568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:01.117925 18568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:58:01.117938 18568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:01.124168 18568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39953
I20260928 04:58:01.124258 18696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39953 every 8 connection(s)
I20260928 04:58:01.124595 18568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:58:01 dist-test-slave-m39l krb5kdc[18461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:58:01.132906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 18568
I20260928 04:58:01.133037 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:58:01.133271 392 external_mini_cluster.cc:1428] Setting key 90c28c34ee6802bc9470895682563ecb
I20260928 04:58:01.135488 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.125429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54945 (local address 127.0.98.62:45521)
0928 04:58:01.125487 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:01.125489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:01.125710 (+ 221us) server_negotiation.cc:407] Connection header received
0928 04:58:01.126950 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:01.126952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:01.126964 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:01.127012 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:01.127758 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.128287 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.129296 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.129495 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.129547 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.131948 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:01.131950 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:01.131950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:01.131957 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:01.133744 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:01.134313 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:01.134314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:01.134315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:01.134381 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:01.134685 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:01.134686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:01.134686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:01.134845 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:01.134918 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:01.135230 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 04:58:01.135348 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 04:58:01.135772 18699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.125351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45521 (local address 127.0.98.1:54945)
0928 04:58:01.125575 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:01.125624 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:01.126792 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:01.127076 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:01.127121 (+ 45us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:01.127412 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:01.127626 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:01.127630 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.128372 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.128373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:01.129217 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:01.129219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.129571 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.129572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:01.129688 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.130283 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:01.130291 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:01.131833 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:01.133824 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:01.133835 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:01.133837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:01.134233 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:01.134449 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:01.134449 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:01.134449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:01.134625 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:01.134912 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:01.134917 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:01.135046 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 04:58:01.135169 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 04:58:01.135338 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260928 04:58:01.136298 18697 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45521
I20260928 04:58:01.136423 18697 heartbeater.cc:464] Registering TS with master...
I20260928 04:58:01.136618 18697 heartbeater.cc:511] Master 127.0.98.62:45521 requested a full tablet report, sending...
I20260928 04:58:01.137218 18506 ts_manager.cc:194] Registered new tserver with Master: 12ad1286a81948489805dcde77cee765 (127.0.98.1:39953)
I20260928 04:58:01.138212 18506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54945
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:58:01.163038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45521
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:01.246906 18702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:01.247148 18702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:01.247174 18702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:01.248334 18702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:01.248377 18702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:01.248406 18702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:58:01.249924 18702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:01.250164 18702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:01.250373 18702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:01.255645 18707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:01.255748 18702 server_base.cc:1034] running on GCE node
W20260928 04:58:01.255713 18710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:01.255869 18708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:01.256177 18702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:01.256423 18702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:01.257583 18702 hybrid_clock.cc:648] HybridClock initialized: now 1790571481257562 us; error 40 us; skew 500 ppm
Sep 28 04:58:01 dist-test-slave-m39l krb5kdc[18461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:01.260366 18702 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:58:01.260916 18702 webserver.cc:460] Webserver started at http://127.0.98.2:46687/ using document root <none> and password file <none>
I20260928 04:58:01.261174 18702 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:01.261245 18702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:01.261391 18702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:01.262570 18702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "c9d5908657184403992d85e4f572b9df"
format_stamp: "Formatted at 2026-09-28 04:58:01 on dist-test-slave-m39l"
server_key: "fe3c6730d791c198a1a17cbe7559f878"
server_key_iv: "3706f1b40d03af91c444ecc23d580bed"
server_key_version: "encryptionkey@0"
I20260928 04:58:01.262950 18702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c9d5908657184403992d85e4f572b9df"
format_stamp: "Formatted at 2026-09-28 04:58:01 on dist-test-slave-m39l"
server_key: "fe3c6730d791c198a1a17cbe7559f878"
server_key_iv: "3706f1b40d03af91c444ecc23d580bed"
server_key_version: "encryptionkey@0"
I20260928 04:58:01.264434 18702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:58:01.265414 18717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:01.265658 18702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:58:01.265772 18702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "c9d5908657184403992d85e4f572b9df"
format_stamp: "Formatted at 2026-09-28 04:58:01 on dist-test-slave-m39l"
server_key: "fe3c6730d791c198a1a17cbe7559f878"
server_key_iv: "3706f1b40d03af91c444ecc23d580bed"
server_key_version: "encryptionkey@0"
I20260928 04:58:01.265859 18702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:01.278844 18702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:01.281275 18702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:01.281448 18702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:01.281703 18702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:58:01.282035 18702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:58:01.282088 18702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:01.282130 18702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:58:01.282174 18702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:01.288573 18702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41845
I20260928 04:58:01.288610 18830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41845 every 8 connection(s)
I20260928 04:58:01.288960 18702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:58:01 dist-test-slave-m39l krb5kdc[18461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:58:01.298514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 18702
I20260928 04:58:01.298655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:58:01.298882 392 external_mini_cluster.cc:1428] Setting key d4164d1afdbbebb28b8b56945f73d252
I20260928 04:58:01.299985 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.289729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48777 (local address 127.0.98.62:45521)
0928 04:58:01.289793 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:01.289795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:01.290010 (+ 215us) server_negotiation.cc:407] Connection header received
0928 04:58:01.291119 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:01.291121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:01.291133 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:01.291192 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:01.291880 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.292410 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.293466 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.293725 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.293799 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.296458 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:01.296460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:01.296460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:01.296467 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:01.298217 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:01.298823 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:01.298825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:01.298825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:01.298895 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:01.299249 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:01.299251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:01.299251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:01.299408 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:01.299477 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:01.299832 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 04:58:01.299916 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:58:01.300379 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.289720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45521 (local address 127.0.98.2:48777)
0928 04:58:01.289883 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:01.289922 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:01.290999 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:01.291276 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:01.291279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:01.291568 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:01.291773 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:01.291777 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.292517 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.292518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:01.293366 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:01.293369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.293824 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.293825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:01.293961 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.294601 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:01.294610 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:01.296343 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:01.298324 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:01.298335 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:01.298338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:01.298730 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:01.298990 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:01.298991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:01.298992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:01.299168 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:01.299473 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:01.299478 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:01.299692 (+ 214us) client_negotiation.cc:769] Sending connection context
0928 04:58:01.299809 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 04:58:01.300014 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260928 04:58:01.300971 18831 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45521
I20260928 04:58:01.301152 18831 heartbeater.cc:464] Registering TS with master...
I20260928 04:58:01.301373 18831 heartbeater.cc:511] Master 127.0.98.62:45521 requested a full tablet report, sending...
I20260928 04:58:01.301795 18506 ts_manager.cc:194] Registered new tserver with Master: c9d5908657184403992d85e4f572b9df (127.0.98.2:41845)
I20260928 04:58:01.302282 18506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48777
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:58:01.326387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45521
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:01.411451 18836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:01.411691 18836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:01.411720 18836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:01.413044 18836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:01.413115 18836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:01.413163 18836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:58:01.415211 18836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:01.415539 18836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:01.415802 18836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:01.421871 18844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:01.421875 18841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:01.421970 18836 server_base.cc:1034] running on GCE node
W20260928 04:58:01.422060 18842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:01.422361 18836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:01.422689 18836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:01.423882 18836 hybrid_clock.cc:648] HybridClock initialized: now 1790571481423836 us; error 71 us; skew 500 ppm
Sep 28 04:58:01 dist-test-slave-m39l krb5kdc[18461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:01.431617 18836 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:58:01.432229 18836 webserver.cc:460] Webserver started at http://127.0.98.3:36755/ using document root <none> and password file <none>
I20260928 04:58:01.432492 18836 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:01.432562 18836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:01.432719 18836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:01.433907 18836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "43c0fbda785e44b7ae8435d2f6c22df7"
format_stamp: "Formatted at 2026-09-28 04:58:01 on dist-test-slave-m39l"
server_key: "d6708ea8619740636295ebed296122d7"
server_key_iv: "c5aa11806c2131ba2524822cd9f0903b"
server_key_version: "encryptionkey@0"
I20260928 04:58:01.434283 18836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "43c0fbda785e44b7ae8435d2f6c22df7"
format_stamp: "Formatted at 2026-09-28 04:58:01 on dist-test-slave-m39l"
server_key: "d6708ea8619740636295ebed296122d7"
server_key_iv: "c5aa11806c2131ba2524822cd9f0903b"
server_key_version: "encryptionkey@0"
I20260928 04:58:01.435804 18836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 04:58:01.436599 18851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:01.436796 18836 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:01.436899 18836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "43c0fbda785e44b7ae8435d2f6c22df7"
format_stamp: "Formatted at 2026-09-28 04:58:01 on dist-test-slave-m39l"
server_key: "d6708ea8619740636295ebed296122d7"
server_key_iv: "c5aa11806c2131ba2524822cd9f0903b"
server_key_version: "encryptionkey@0"
I20260928 04:58:01.436975 18836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:01.454675 18836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:01.456876 18836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:01.457063 18836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:01.457324 18836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:58:01.457634 18836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:58:01.457688 18836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:01.457731 18836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:58:01.457760 18836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:01.464071 18836 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43949
I20260928 04:58:01.464105 18964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43949 every 8 connection(s)
I20260928 04:58:01.464452 18836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:58:01 dist-test-slave-m39l krb5kdc[18461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:58:01.472294 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 18836
I20260928 04:58:01.472404 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:58:01.472573 392 external_mini_cluster.cc:1428] Setting key fc5aa4824bbd6a4948bfc1c7034b08fd
I20260928 04:58:01.475349 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.465191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48961 (local address 127.0.98.62:45521)
0928 04:58:01.465278 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:01.465281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:01.465515 (+ 234us) server_negotiation.cc:407] Connection header received
0928 04:58:01.466715 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:01.466717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:01.466727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:01.466773 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:01.467538 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.468038 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.469419 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.469612 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.469676 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.472140 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:01.472142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:01.472142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:01.472149 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:01.473889 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:01.474348 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:01.474348 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:01.474349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:01.474399 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:01.474688 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:01.474689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:01.474689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:01.474799 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:01.474852 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:01.475213 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 04:58:01.475276 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 04:58:01.475587 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.465152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45521 (local address 127.0.98.3:48961)
0928 04:58:01.465363 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:01.465405 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:01.466584 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:01.466852 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:01.466856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:01.467226 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:01.467449 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:01.467454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.468148 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.468150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:01.469330 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:01.469333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.469697 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.469698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:01.469818 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.470346 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:01.470356 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:01.472024 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:01.474025 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:01.474031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:01.474033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:01.474281 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:01.474454 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:01.474454 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:01.474455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:01.474631 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:01.474885 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:01.474891 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:01.475076 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 04:58:01.475176 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 04:58:01.475312 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":102,"threads_started":1}
I20260928 04:58:01.476054 18965 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45521
I20260928 04:58:01.476166 18965 heartbeater.cc:464] Registering TS with master...
I20260928 04:58:01.476361 18965 heartbeater.cc:511] Master 127.0.98.62:45521 requested a full tablet report, sending...
I20260928 04:58:01.476716 18506 ts_manager.cc:194] Registered new tserver with Master: 43c0fbda785e44b7ae8435d2f6c22df7 (127.0.98.3:43949)
I20260928 04:58:01.477111 18506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48961
I20260928 04:58:01.484498 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:58:01.492882 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.486022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53804 (local address 127.0.98.62:45521)
0928 04:58:01.486110 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:01.486112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:01.486233 (+ 121us) server_negotiation.cc:407] Connection header received
0928 04:58:01.486326 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:01.486327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:01.486338 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:01.486384 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:01.487035 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.487563 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.488520 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.488741 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.488824 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.490258 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:01.490260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:01.490261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:01.490269 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:01.491581 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:01.492023 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:01.492025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:01.492025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:01.492085 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:01.492301 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:01.492302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:01.492303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:01.492458 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:01.492575 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:01.492703 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 04:58:01.492772 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 04:58:01.494772 18506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:58:01.496397 18506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:58:01.503896 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.499446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43949 (local address 127.0.0.1:56362)
0928 04:58:01.500102 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:01.500113 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:01.500158 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:01.500420 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:01.500421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:01.500422 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:01.500456 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:01.500458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.501439 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.501440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:01.502801 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:01.502805 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.503611 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.503613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:01.503694 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.503708 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 04:58:01.503777 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 04:58:01.503823 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":65,"threads_started":1}
I20260928 04:58:01.503937 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.499915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56362 (local address 127.0.98.3:43949)
0928 04:58:01.500272 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:01.500277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:01.500286 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:58:01.500325 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:01.500328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:01.500342 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:01.500413 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:01.500525 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.501329 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.502898 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.503504 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.503580 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.503649 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:01.503804 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 04:58:01.503847 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":96,"threads_started":1}
I20260928 04:58:01.504482 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.499200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41845 (local address 127.0.0.1:54090)
0928 04:58:01.499355 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:01.499403 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:01.499472 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:01.499883 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:01.499887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:01.499889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:01.499992 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:01.499995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.501651 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.501652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:01.503236 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:01.503240 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.504206 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.504207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:01.504280 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.504284 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:58:01.504365 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 04:58:01.504409 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":37,"thread_start_us":93,"threads_started":1}
I20260928 04:58:01.504772 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.499339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54090 (local address 127.0.98.2:41845)
0928 04:58:01.499568 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:01.499574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:01.499588 (+ 14us) server_negotiation.cc:407] Connection header received
0928 04:58:01.499635 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:01.499638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:01.499655 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:01.499770 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:01.500127 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.501487 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.503356 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.504130 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.504204 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.504259 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:01.504398 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 04:58:01.504449 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
I20260928 04:58:01.504768 18899 tablet_service.cc:1467] Processing CreateTablet for tablet d662aacb3df746cba9978a17dd4f55fc (DEFAULT_TABLE table=test-table [id=6a9d1ddbebff43e6a2bd83c9815d2e8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:58:01.505205 18899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d662aacb3df746cba9978a17dd4f55fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:01.505452 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.499202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39953 (local address 127.0.0.1:43030)
0928 04:58:01.499400 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:01.499441 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:01.499504 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:01.500206 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:01.500207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:01.500209 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:01.500258 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:01.500260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.501635 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.501636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:01.503237 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:01.503240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.505212 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.505214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:01.505282 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.505286 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:58:01.505338 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 04:58:01.505384 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":76,"thread_start_us":79,"threads_started":1}
I20260928 04:58:01.505465 18765 tablet_service.cc:1467] Processing CreateTablet for tablet d662aacb3df746cba9978a17dd4f55fc (DEFAULT_TABLE table=test-table [id=6a9d1ddbebff43e6a2bd83c9815d2e8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:58:01.505709 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.499594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43030 (local address 127.0.98.1:39953)
0928 04:58:01.500022 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:01.500027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:01.500039 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:58:01.500088 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:01.500091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:01.500108 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:01.500198 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:01.500328 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.501523 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.504306 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.505041 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.505116 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.505172 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:01.505573 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 04:58:01.505628 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":140,"threads_started":1}
I20260928 04:58:01.505805 18765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d662aacb3df746cba9978a17dd4f55fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:01.506350 18631 tablet_service.cc:1467] Processing CreateTablet for tablet d662aacb3df746cba9978a17dd4f55fc (DEFAULT_TABLE table=test-table [id=6a9d1ddbebff43e6a2bd83c9815d2e8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:58:01.506734 18631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d662aacb3df746cba9978a17dd4f55fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:01.508172 18982 tablet_bootstrap.cc:492] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7: Bootstrap starting.
I20260928 04:58:01.508880 18982 tablet_bootstrap.cc:654] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:01.509334 18983 tablet_bootstrap.cc:492] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df: Bootstrap starting.
I20260928 04:58:01.509408 18982 log.cc:824] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:01.509977 18983 tablet_bootstrap.cc:654] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:01.510278 18982 tablet_bootstrap.cc:492] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7: No bootstrap required, opened a new log
I20260928 04:58:01.510316 18985 tablet_bootstrap.cc:492] T d662aacb3df746cba9978a17dd4f55fc P 12ad1286a81948489805dcde77cee765: Bootstrap starting.
I20260928 04:58:01.510581 18982 ts_tablet_manager.cc:1397] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:58:01.510635 18983 log.cc:824] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:01.511034 18985 tablet_bootstrap.cc:654] T d662aacb3df746cba9978a17dd4f55fc P 12ad1286a81948489805dcde77cee765: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:01.511541 18983 tablet_bootstrap.cc:492] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df: No bootstrap required, opened a new log
I20260928 04:58:01.511627 18983 ts_tablet_manager.cc:1397] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:58:01.511647 18985 log.cc:824] T d662aacb3df746cba9978a17dd4f55fc P 12ad1286a81948489805dcde77cee765: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:01.512472 18985 tablet_bootstrap.cc:492] T d662aacb3df746cba9978a17dd4f55fc P 12ad1286a81948489805dcde77cee765: No bootstrap required, opened a new log
I20260928 04:58:01.512550 18985 ts_tablet_manager.cc:1397] T d662aacb3df746cba9978a17dd4f55fc P 12ad1286a81948489805dcde77cee765: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:58:01.512606 18982 raft_consensus.cc:348] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d5908657184403992d85e4f572b9df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41845 } } peers { permanent_uuid: "43c0fbda785e44b7ae8435d2f6c22df7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43949 } } peers { permanent_uuid: "12ad1286a81948489805dcde77cee765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39953 } }
I20260928 04:58:01.512795 18982 raft_consensus.cc:374] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:01.512838 18982 raft_consensus.cc:729] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c0fbda785e44b7ae8435d2f6c22df7, State: Initialized, Role: FOLLOWER
I20260928 04:58:01.512959 18982 consensus_queue.cc:260] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d5908657184403992d85e4f572b9df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41845 } } peers { permanent_uuid: "43c0fbda785e44b7ae8435d2f6c22df7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43949 } } peers { permanent_uuid: "12ad1286a81948489805dcde77cee765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39953 } }
I20260928 04:58:01.513237 18982 ts_tablet_manager.cc:1428] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:58:01.513371 18965 heartbeater.cc:503] Master 127.0.98.62:45521 was elected leader, sending a full tablet report...
I20260928 04:58:01.513881 18983 raft_consensus.cc:348] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d5908657184403992d85e4f572b9df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41845 } } peers { permanent_uuid: "43c0fbda785e44b7ae8435d2f6c22df7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43949 } } peers { permanent_uuid: "12ad1286a81948489805dcde77cee765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39953 } }
I20260928 04:58:01.514073 18983 raft_consensus.cc:374] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:01.514117 18983 raft_consensus.cc:729] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d5908657184403992d85e4f572b9df, State: Initialized, Role: FOLLOWER
I20260928 04:58:01.514254 18983 consensus_queue.cc:260] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d5908657184403992d85e4f572b9df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41845 } } peers { permanent_uuid: "43c0fbda785e44b7ae8435d2f6c22df7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43949 } } peers { permanent_uuid: "12ad1286a81948489805dcde77cee765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39953 } }
I20260928 04:58:01.514587 18983 ts_tablet_manager.cc:1428] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:58:01.514662 18831 heartbeater.cc:503] Master 127.0.98.62:45521 was elected leader, sending a full tablet report...
I20260928 04:58:01.514689 18985 raft_consensus.cc:348] T d662aacb3df746cba9978a17dd4f55fc P 12ad1286a81948489805dcde77cee765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d5908657184403992d85e4f572b9df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41845 } } peers { permanent_uuid: "43c0fbda785e44b7ae8435d2f6c22df7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43949 } } peers { permanent_uuid: "12ad1286a81948489805dcde77cee765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39953 } }
I20260928 04:58:01.514886 18985 raft_consensus.cc:374] T d662aacb3df746cba9978a17dd4f55fc P 12ad1286a81948489805dcde77cee765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:01.514931 18985 raft_consensus.cc:729] T d662aacb3df746cba9978a17dd4f55fc P 12ad1286a81948489805dcde77cee765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ad1286a81948489805dcde77cee765, State: Initialized, Role: FOLLOWER
I20260928 04:58:01.515057 18985 consensus_queue.cc:260] T d662aacb3df746cba9978a17dd4f55fc P 12ad1286a81948489805dcde77cee765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d5908657184403992d85e4f572b9df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41845 } } peers { permanent_uuid: "43c0fbda785e44b7ae8435d2f6c22df7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43949 } } peers { permanent_uuid: "12ad1286a81948489805dcde77cee765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39953 } }
I20260928 04:58:01.515376 18985 ts_tablet_manager.cc:1428] T d662aacb3df746cba9978a17dd4f55fc P 12ad1286a81948489805dcde77cee765: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:58:01.515383 18697 heartbeater.cc:503] Master 127.0.98.62:45521 was elected leader, sending a full tablet report...
I20260928 04:58:01.530612 18989 raft_consensus.cc:482] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:58:01.530792 18989 raft_consensus.cc:504] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d5908657184403992d85e4f572b9df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41845 } } peers { permanent_uuid: "43c0fbda785e44b7ae8435d2f6c22df7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43949 } } peers { permanent_uuid: "12ad1286a81948489805dcde77cee765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39953 } }
I20260928 04:58:01.531257 18989 leader_election.cc:288] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43c0fbda785e44b7ae8435d2f6c22df7 (127.0.98.3:43949), 12ad1286a81948489805dcde77cee765 (127.0.98.1:39953)
I20260928 04:58:01.534291 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.531562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56865 (local address 127.0.98.3:43949)
0928 04:58:01.531621 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:01.531623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:01.531750 (+ 127us) server_negotiation.cc:407] Connection header received
0928 04:58:01.531849 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:01.531851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:01.531862 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:01.531922 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:01.532065 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.532554 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.533534 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.533914 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.533994 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.534006 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:01.534165 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 04:58:01.534212 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 04:58:01.534298 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.531427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39953 (local address 127.0.98.2:48037)
0928 04:58:01.531472 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:01.531485 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:01.531535 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:01.531738 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:01.531739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:01.531742 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:01.531815 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:01.531818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.532541 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.532542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:01.533434 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:01.533437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.534022 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.534023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:01.534117 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.534135 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 04:58:01.534193 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 04:58:01.534231 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 04:58:01.534324 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.531521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48037 (local address 127.0.98.1:39953)
0928 04:58:01.531567 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:01.531569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:01.531616 (+ 47us) server_negotiation.cc:407] Connection header received
0928 04:58:01.531645 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:01.531646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:01.531660 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:01.531716 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:01.531884 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.532455 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.533537 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.533920 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.534006 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.534016 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:01.534210 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 04:58:01.534256 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:58:01.534301 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.531435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43949 (local address 127.0.98.2:56865)
0928 04:58:01.531698 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:01.531708 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:01.531772 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:01.531937 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:01.531937 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:01.531939 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:01.532001 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:01.532003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.532633 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.532634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:01.533435 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:01.533437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.533997 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.533998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:01.534073 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.534091 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 04:58:01.534138 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 04:58:01.534193 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":24,"thread_start_us":124,"threads_started":1}
I20260928 04:58:01.534713 18919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d662aacb3df746cba9978a17dd4f55fc" candidate_uuid: "c9d5908657184403992d85e4f572b9df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c0fbda785e44b7ae8435d2f6c22df7" is_pre_election: true
I20260928 04:58:01.534781 18651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d662aacb3df746cba9978a17dd4f55fc" candidate_uuid: "c9d5908657184403992d85e4f572b9df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ad1286a81948489805dcde77cee765" is_pre_election: true
I20260928 04:58:01.534909 18919 raft_consensus.cc:2454] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9d5908657184403992d85e4f572b9df in term 0.
I20260928 04:58:01.534958 18651 raft_consensus.cc:2454] T d662aacb3df746cba9978a17dd4f55fc P 12ad1286a81948489805dcde77cee765 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9d5908657184403992d85e4f572b9df in term 0.
I20260928 04:58:01.535153 18720 leader_election.cc:302] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c0fbda785e44b7ae8435d2f6c22df7, c9d5908657184403992d85e4f572b9df; no voters:
I20260928 04:58:01.535308 18989 raft_consensus.cc:2781] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:58:01.535363 18989 raft_consensus.cc:482] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:58:01.535387 18989 raft_consensus.cc:3037] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:01.536070 18989 raft_consensus.cc:504] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d5908657184403992d85e4f572b9df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41845 } } peers { permanent_uuid: "43c0fbda785e44b7ae8435d2f6c22df7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43949 } } peers { permanent_uuid: "12ad1286a81948489805dcde77cee765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39953 } }
I20260928 04:58:01.536232 18989 leader_election.cc:288] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [CANDIDATE]: Term 1 election: Requested vote from peers 43c0fbda785e44b7ae8435d2f6c22df7 (127.0.98.3:43949), 12ad1286a81948489805dcde77cee765 (127.0.98.1:39953)
I20260928 04:58:01.536412 18919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d662aacb3df746cba9978a17dd4f55fc" candidate_uuid: "c9d5908657184403992d85e4f572b9df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c0fbda785e44b7ae8435d2f6c22df7"
I20260928 04:58:01.536425 18651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d662aacb3df746cba9978a17dd4f55fc" candidate_uuid: "c9d5908657184403992d85e4f572b9df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ad1286a81948489805dcde77cee765"
I20260928 04:58:01.536512 18651 raft_consensus.cc:3037] T d662aacb3df746cba9978a17dd4f55fc P 12ad1286a81948489805dcde77cee765 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:01.536512 18919 raft_consensus.cc:3037] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:01.537448 18651 raft_consensus.cc:2454] T d662aacb3df746cba9978a17dd4f55fc P 12ad1286a81948489805dcde77cee765 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9d5908657184403992d85e4f572b9df in term 1.
I20260928 04:58:01.537448 18919 raft_consensus.cc:2454] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9d5908657184403992d85e4f572b9df in term 1.
I20260928 04:58:01.537690 18720 leader_election.cc:302] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c0fbda785e44b7ae8435d2f6c22df7, c9d5908657184403992d85e4f572b9df; no voters:
I20260928 04:58:01.537817 18989 raft_consensus.cc:2781] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:01.537997 18989 raft_consensus.cc:686] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [term 1 LEADER]: Becoming Leader. State: Replica: c9d5908657184403992d85e4f572b9df, State: Running, Role: LEADER
I20260928 04:58:01.538110 18989 consensus_queue.cc:237] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d5908657184403992d85e4f572b9df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41845 } } peers { permanent_uuid: "43c0fbda785e44b7ae8435d2f6c22df7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43949 } } peers { permanent_uuid: "12ad1286a81948489805dcde77cee765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39953 } }
I20260928 04:58:01.539183 18506 catalog_manager.cc:5495] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df reported cstate change: term changed from 0 to 1, leader changed from <none> to c9d5908657184403992d85e4f572b9df (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c9d5908657184403992d85e4f572b9df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d5908657184403992d85e4f572b9df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43c0fbda785e44b7ae8435d2f6c22df7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12ad1286a81948489805dcde77cee765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39953 } health_report { overall_health: UNKNOWN } } }
W20260928 04:58:01.539539 18832 tablet.cc:2369] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:58:01.555236 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.552489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.0.98.2:41845)
0928 04:58:01.552551 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:01.552558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:01.552589 (+ 31us) server_negotiation.cc:407] Connection header received
0928 04:58:01.552626 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:01.552627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:01.552648 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:01.552713 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:58:01.552844 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.553377 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.554182 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.554403 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.554620 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.554769 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:58:01.554866 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:58:01.554951 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:01.555130 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 04:58:01.555142 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:58:01.557399 18919 raft_consensus.cc:1261] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7 [term 1 FOLLOWER]: Refusing update from remote peer c9d5908657184403992d85e4f572b9df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:58:01.557399 18651 raft_consensus.cc:1261] T d662aacb3df746cba9978a17dd4f55fc P 12ad1286a81948489805dcde77cee765 [term 1 FOLLOWER]: Refusing update from remote peer c9d5908657184403992d85e4f572b9df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:58:01.557834 18989 consensus_queue.cc:1035] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [LEADER]: Connected to new peer: Peer: permanent_uuid: "43c0fbda785e44b7ae8435d2f6c22df7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:58:01.557997 18992 consensus_queue.cc:1035] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ad1286a81948489805dcde77cee765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:58:01.560635 18995 mvcc.cc:205] Tried to move back new op lower bound from 7334180788455886848 to 7334180788381007872. Current Snapshot: MvccSnapshot[applied={T|T < 7334180788455886848}]
I20260928 04:58:01.560950 18997 mvcc.cc:205] Tried to move back new op lower bound from 7334180788455886848 to 7334180788381007872. Current Snapshot: MvccSnapshot[applied={T|T < 7334180788455886848}]
I20260928 04:58:01.563655 18505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:58:01.563786 18505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:58:01.564848 18505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b29934b42b3242fc91ccba9f41e2d7fe: Sending DeleteTablet for 3 replicas of tablet d662aacb3df746cba9978a17dd4f55fc
I20260928 04:58:01.565313 18765 tablet_service.cc:1514] Processing DeleteTablet for tablet d662aacb3df746cba9978a17dd4f55fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:58:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54090
I20260928 04:58:01.565363 18899 tablet_service.cc:1514] Processing DeleteTablet for tablet d662aacb3df746cba9978a17dd4f55fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:58:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56362
I20260928 04:58:01.565351 18631 tablet_service.cc:1514] Processing DeleteTablet for tablet d662aacb3df746cba9978a17dd4f55fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:58:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43030
I20260928 04:58:01.565659 19006 tablet_replica.cc:330] T d662aacb3df746cba9978a17dd4f55fc P 12ad1286a81948489805dcde77cee765: stopping tablet replica
I20260928 04:58:01.565881 19004 tablet_replica.cc:330] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df: stopping tablet replica
I20260928 04:58:01.565886 19006 raft_consensus.cc:2229] T d662aacb3df746cba9978a17dd4f55fc P 12ad1286a81948489805dcde77cee765 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:58:01.566038 19004 raft_consensus.cc:2229] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:58:01.566057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 18568
I20260928 04:58:01.566103 19006 raft_consensus.cc:2258] T d662aacb3df746cba9978a17dd4f55fc P 12ad1286a81948489805dcde77cee765 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:58:01.567001 19005 tablet_replica.cc:330] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7: stopping tablet replica
I20260928 04:58:01.567121 19005 raft_consensus.cc:2229] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:58:01.567252 19005 raft_consensus.cc:2258] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:58:01.567857 19005 ts_tablet_manager.cc:1905] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:58:01.570788 19005 ts_tablet_manager.cc:1918] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:58:01.570894 19005 log.cc:1196] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/d662aacb3df746cba9978a17dd4f55fc
I20260928 04:58:01.571040 19004 raft_consensus.cc:2258] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:58:01.571197 19005 ts_tablet_manager.cc:1939] T d662aacb3df746cba9978a17dd4f55fc P 43c0fbda785e44b7ae8435d2f6c22df7: Deleting consensus metadata
I20260928 04:58:01.571691 19004 ts_tablet_manager.cc:1905] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:58:01.572561 18492 catalog_manager.cc:4841] TS 43c0fbda785e44b7ae8435d2f6c22df7 (127.0.98.3:43949): tablet d662aacb3df746cba9978a17dd4f55fc (table test-table [id=6a9d1ddbebff43e6a2bd83c9815d2e8c]) successfully deleted
I20260928 04:58:01.573452 19004 ts_tablet_manager.cc:1918] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:58:01.573527 19004 log.cc:1196] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/d662aacb3df746cba9978a17dd4f55fc
I20260928 04:58:01.573837 19004 ts_tablet_manager.cc:1939] T d662aacb3df746cba9978a17dd4f55fc P c9d5908657184403992d85e4f572b9df: Deleting consensus metadata
W20260928 04:58:01.574127 18491 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39953 (error 108)
I20260928 04:58:01.574425 18492 catalog_manager.cc:4841] TS c9d5908657184403992d85e4f572b9df (127.0.98.2:41845): tablet d662aacb3df746cba9978a17dd4f55fc (table test-table [id=6a9d1ddbebff43e6a2bd83c9815d2e8c]) successfully deleted
W20260928 04:58:01.574924 18976 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:01.574759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39953 (local address 127.0.0.1:43046)
0928 04:58:01.574825 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:01.574875 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39953: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 04:58:01.575075 18491 catalog_manager.cc:4584] TS 12ad1286a81948489805dcde77cee765 (127.0.98.1:39953): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d662aacb3df746cba9978a17dd4f55fc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39953: connect: Connection reset by peer (error 104)
I20260928 04:58:01.575331 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 18702
I20260928 04:58:01.580304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 18836
I20260928 04:58:01.585029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 18474
2026-09-28T04:58:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2074 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T04:58:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:01Z Disabled control of system clock
I20260928 04:58:01.615741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45791
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:58:01.700290 19011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:01.700547 19011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:01.700572 19011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:01.701795 19011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:01.701845 19011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:01.701858 19011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:01.701871 19011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:01.701884 19011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:01.703299 19011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:01.703538 19011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:01.703801 19011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:01.708902 19017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:01.708948 19019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:01.709163 19011 server_base.cc:1034] running on GCE node
W20260928 04:58:01.708901 19016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:01.709600 19011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:01.709921 19011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:01.711090 19011 hybrid_clock.cc:648] HybridClock initialized: now 1790571481711066 us; error 43 us; skew 500 ppm
I20260928 04:58:01.712704 19011 webserver.cc:460] Webserver started at http://127.0.98.62:39047/ using document root <none> and password file <none>
I20260928 04:58:01.713135 19011 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:01.713225 19011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:01.713377 19011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:01.714671 19011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "2d3f066ed70641d88426c3c503a1fb0f"
format_stamp: "Formatted at 2026-09-28 04:58:01 on dist-test-slave-m39l"
server_key: "c432ea6654c752498c3a182c5c9dae32"
server_key_iv: "260893daf7da07a982e25339259722a8"
server_key_version: "encryptionkey@0"
I20260928 04:58:01.715104 19011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "2d3f066ed70641d88426c3c503a1fb0f"
format_stamp: "Formatted at 2026-09-28 04:58:01 on dist-test-slave-m39l"
server_key: "c432ea6654c752498c3a182c5c9dae32"
server_key_iv: "260893daf7da07a982e25339259722a8"
server_key_version: "encryptionkey@0"
I20260928 04:58:01.716578 19011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:58:01.717500 19025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:01.717731 19011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:01.717832 19011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "2d3f066ed70641d88426c3c503a1fb0f"
format_stamp: "Formatted at 2026-09-28 04:58:01 on dist-test-slave-m39l"
server_key: "c432ea6654c752498c3a182c5c9dae32"
server_key_iv: "260893daf7da07a982e25339259722a8"
server_key_version: "encryptionkey@0"
I20260928 04:58:01.717917 19011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:01.731725 19011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:01.732091 19011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:01.732252 19011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:01.736305 19011 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45791
I20260928 04:58:01.736343 19077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45791 every 8 connection(s)
I20260928 04:58:01.736714 19011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:01.737835 19078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:01.740345 19078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f: Bootstrap starting.
I20260928 04:58:01.740952 19078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:01.741010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 19011
I20260928 04:58:01.741155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:01.741317 19078 log.cc:824] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:01.741362 392 external_mini_cluster.cc:1428] Setting key ee18c04c7eed7863a610320676b78418
I20260928 04:58:01.742162 19078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f: No bootstrap required, opened a new log
I20260928 04:58:01.744952 19078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3f066ed70641d88426c3c503a1fb0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260928 04:58:01.745100 19078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:01.745128 19078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3f066ed70641d88426c3c503a1fb0f, State: Initialized, Role: FOLLOWER
I20260928 04:58:01.745261 19078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3f066ed70641d88426c3c503a1fb0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260928 04:58:01.745333 19078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:01.745363 19078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:01.745410 19078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:01.746165 19078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3f066ed70641d88426c3c503a1fb0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260928 04:58:01.746279 19078 leader_election.cc:302] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d3f066ed70641d88426c3c503a1fb0f; no voters:
I20260928 04:58:01.746474 19078 leader_election.cc:288] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:01.746584 19083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:01.746764 19083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f [term 1 LEADER]: Becoming Leader. State: Replica: 2d3f066ed70641d88426c3c503a1fb0f, State: Running, Role: LEADER
I20260928 04:58:01.746793 19078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:01.746937 19083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3f066ed70641d88426c3c503a1fb0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260928 04:58:01.747305 19083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d3f066ed70641d88426c3c503a1fb0f. Latest consensus state: current_term: 1 leader_uuid: "2d3f066ed70641d88426c3c503a1fb0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3f066ed70641d88426c3c503a1fb0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } } }
I20260928 04:58:01.747390 19083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:01.747331 19085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d3f066ed70641d88426c3c503a1fb0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3f066ed70641d88426c3c503a1fb0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } } }
I20260928 04:58:01.747452 19085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:01.747598 19092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:01.748463 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.741885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46784 (local address 127.0.98.62:45791)
0928 04:58:01.742101 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:01.742110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:01.742142 (+ 32us) server_negotiation.cc:407] Connection header received
0928 04:58:01.742742 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:01.742753 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:01.742837 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:01.743010 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:01.743254 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.744748 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.745744 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.746420 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.746607 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.747610 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:01.747614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:01.747615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:01.747859 (+ 244us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:01.747911 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:01.747927 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:01.747973 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:01.748100 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 04:58:01.748199 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260928 04:58:01.748219 19092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:01.750221 19092 catalog_manager.cc:1347] Generated new cluster ID: 3ef8b8baf8344a26a164d0fd5d7ad544
I20260928 04:58:01.750273 19092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:58:01.758831 19092 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:01.759459 19092 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:01.766788 19092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f: Generated new TSK 0
I20260928 04:58:01.767037 19092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:01.771119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45791
--builtin_ntp_servers=127.0.98.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:58:01.863716 19102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:01.864012 19102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:01.864059 19102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:01.865499 19102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:01.865583 19102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:01.865648 19102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:58:01.867449 19102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:01.867779 19102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:01.868152 19102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:01.874210 19108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:01.874234 19110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:01.874210 19107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:01.874990 19102 server_base.cc:1034] running on GCE node
I20260928 04:58:01.875284 19102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:01.875618 19102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:01.876797 19102 hybrid_clock.cc:648] HybridClock initialized: now 1790571481876785 us; error 32 us; skew 500 ppm
I20260928 04:58:01.878298 19102 webserver.cc:460] Webserver started at http://127.0.98.1:41359/ using document root <none> and password file <none>
I20260928 04:58:01.878671 19102 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:01.878752 19102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:01.878913 19102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:01.880218 19102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "eb9520a3161e4a938d4f566112feb5e9"
format_stamp: "Formatted at 2026-09-28 04:58:01 on dist-test-slave-m39l"
server_key: "762cd290fe5134984cc26e821e876478"
server_key_iv: "f305e635ec8cf82b450832771675742c"
server_key_version: "encryptionkey@0"
I20260928 04:58:01.880645 19102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eb9520a3161e4a938d4f566112feb5e9"
format_stamp: "Formatted at 2026-09-28 04:58:01 on dist-test-slave-m39l"
server_key: "762cd290fe5134984cc26e821e876478"
server_key_iv: "f305e635ec8cf82b450832771675742c"
server_key_version: "encryptionkey@0"
I20260928 04:58:01.882254 19102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:58:01.883332 19116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:01.883595 19102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:58:01.883700 19102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "eb9520a3161e4a938d4f566112feb5e9"
format_stamp: "Formatted at 2026-09-28 04:58:01 on dist-test-slave-m39l"
server_key: "762cd290fe5134984cc26e821e876478"
server_key_iv: "f305e635ec8cf82b450832771675742c"
server_key_version: "encryptionkey@0"
I20260928 04:58:01.883793 19102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:01.897233 19102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:01.897619 19102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:01.897791 19102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:01.898080 19102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:58:01.898432 19102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:58:01.898483 19102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:01.898559 19102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:58:01.898586 19102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:01.904734 19102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39999
I20260928 04:58:01.904785 19229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39999 every 8 connection(s)
I20260928 04:58:01.905251 19102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:58:01.906610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 19102
I20260928 04:58:01.906741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:58:01.906932 392 external_mini_cluster.cc:1428] Setting key 5c06f8bad47b1eb266e844a834ad4e52
I20260928 04:58:01.908030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45791
--builtin_ntp_servers=127.0.98.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 04:58:01.911360 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.906071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50009 (local address 127.0.98.62:45791)
0928 04:58:01.906192 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:01.906194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:01.906395 (+ 201us) server_negotiation.cc:407] Connection header received
0928 04:58:01.907122 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:01.907124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:01.907137 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:01.907202 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:01.908037 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.908603 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.909644 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.909847 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.909935 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.911029 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:01.911031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:01.911031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:01.911037 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:01.911056 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:01.911060 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:01.911115 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:01.911238 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 04:58:01.911276 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 04:58:01.911646 19232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:01.906052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:50009)
0928 04:58:01.906260 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:01.906304 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:01.907000 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:01.907283 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:01.907288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:01.907299 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:01.907926 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:01.907932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.908706 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.908708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:01.909535 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:01.909538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:01.909952 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:01.909953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:01.910109 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:01.910846 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:58:01.910847 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:01.910890 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:58:01.910924 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:01.911131 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:01.911137 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:01.911141 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:58:01.911213 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 04:58:01.911346 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260928 04:58:01.912223 19230 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45791
I20260928 04:58:01.912353 19230 heartbeater.cc:464] Registering TS with master...
I20260928 04:58:01.912583 19230 heartbeater.cc:511] Master 127.0.98.62:45791 requested a full tablet report, sending...
I20260928 04:58:01.913348 19042 ts_manager.cc:194] Registered new tserver with Master: eb9520a3161e4a938d4f566112feb5e9 (127.0.98.1:39999)
I20260928 04:58:01.914088 19042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50009
W20260928 04:58:01.992501 19233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:01.992748 19233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:01.992774 19233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:01.994088 19233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:01.994143 19233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:01.994174 19233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:58:01.995800 19233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:01.996100 19233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:01.996433 19233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:02.002269 19239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:02.002269 19241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:02.002271 19238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:02.002630 19233 server_base.cc:1034] running on GCE node
I20260928 04:58:02.002899 19233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:02.003140 19233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:02.004331 19233 hybrid_clock.cc:648] HybridClock initialized: now 1790571482004309 us; error 39 us; skew 500 ppm
I20260928 04:58:02.005802 19233 webserver.cc:460] Webserver started at http://127.0.98.2:42959/ using document root <none> and password file <none>
I20260928 04:58:02.006125 19233 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:02.006230 19233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:02.006405 19233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:02.008234 19233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "86024196741445809d7f1e619194e774"
format_stamp: "Formatted at 2026-09-28 04:58:02 on dist-test-slave-m39l"
server_key: "30534b4b8fe7d524b422285cd7036d6c"
server_key_iv: "c02de8c9c6f121211a572b624567771a"
server_key_version: "encryptionkey@0"
I20260928 04:58:02.008787 19233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "86024196741445809d7f1e619194e774"
format_stamp: "Formatted at 2026-09-28 04:58:02 on dist-test-slave-m39l"
server_key: "30534b4b8fe7d524b422285cd7036d6c"
server_key_iv: "c02de8c9c6f121211a572b624567771a"
server_key_version: "encryptionkey@0"
I20260928 04:58:02.010862 19233 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 04:58:02.011811 19247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:02.012053 19233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:02.012282 19233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "86024196741445809d7f1e619194e774"
format_stamp: "Formatted at 2026-09-28 04:58:02 on dist-test-slave-m39l"
server_key: "30534b4b8fe7d524b422285cd7036d6c"
server_key_iv: "c02de8c9c6f121211a572b624567771a"
server_key_version: "encryptionkey@0"
I20260928 04:58:02.012404 19233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:02.022455 19233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:02.022921 19233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:02.023082 19233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:02.023335 19233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:58:02.023729 19233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:58:02.023790 19233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:02.023835 19233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:58:02.023869 19233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:02.032152 19233 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46327
I20260928 04:58:02.032655 19233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:58:02.033252 19360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46327 every 8 connection(s)
I20260928 04:58:02.038731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 19233
I20260928 04:58:02.038889 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:58:02.039130 392 external_mini_cluster.cc:1428] Setting key 1a796161a5cdff0e9e080276fd294746
I20260928 04:58:02.040298 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45791
--builtin_ntp_servers=127.0.98.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 04:58:02.041522 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.033787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41369 (local address 127.0.98.62:45791)
0928 04:58:02.033839 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:02.033842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:02.033902 (+ 60us) server_negotiation.cc:407] Connection header received
0928 04:58:02.034743 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:02.034745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:02.034760 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:02.034796 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:02.036002 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.036851 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.038531 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.038839 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.038917 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:02.041129 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:02.041131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:02.041132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:02.041139 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:02.041160 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:02.041165 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:02.041197 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:02.041343 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 04:58:02.041383 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 04:58:02.042147 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.033515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:41369)
0928 04:58:02.033757 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:02.033809 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:02.034539 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:02.035103 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:02.035107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:02.035119 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:02.035867 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:02.035874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.036983 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.036984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:02.038371 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:02.038375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.039699 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.039701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:02.039929 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:02.040913 (+ 984us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:58:02.040913 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:02.040964 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:58:02.041002 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:02.041243 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:02.041248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:02.041252 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:58:02.041698 (+ 446us) client_negotiation.cc:240] Negotiation successful
0928 04:58:02.041863 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":94,"threads_started":1}
I20260928 04:58:02.042758 19361 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45791
I20260928 04:58:02.042870 19361 heartbeater.cc:464] Registering TS with master...
I20260928 04:58:02.043143 19361 heartbeater.cc:511] Master 127.0.98.62:45791 requested a full tablet report, sending...
I20260928 04:58:02.043620 19042 ts_manager.cc:194] Registered new tserver with Master: 86024196741445809d7f1e619194e774 (127.0.98.2:46327)
I20260928 04:58:02.044217 19042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41369
W20260928 04:58:02.129501 19364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:02.129761 19364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:02.129789 19364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:02.131189 19364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:02.131243 19364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:02.131270 19364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:58:02.132733 19364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:02.133023 19364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:02.133360 19364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:02.138940 19370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:02.138934 19364 server_base.cc:1034] running on GCE node
W20260928 04:58:02.138934 19372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:02.139675 19369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:02.139947 19364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:02.140236 19364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:02.141422 19364 hybrid_clock.cc:648] HybridClock initialized: now 1790571482141408 us; error 46 us; skew 500 ppm
I20260928 04:58:02.142989 19364 webserver.cc:460] Webserver started at http://127.0.98.3:38621/ using document root <none> and password file <none>
I20260928 04:58:02.143285 19364 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:02.143361 19364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:02.143522 19364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:02.144774 19364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "8338cbf6381f456288658aee30624499"
format_stamp: "Formatted at 2026-09-28 04:58:02 on dist-test-slave-m39l"
server_key: "b4009147d4c13655c98108aa8c4dcba0"
server_key_iv: "307f9b449e3df8decf6675cc6f95849c"
server_key_version: "encryptionkey@0"
I20260928 04:58:02.145186 19364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8338cbf6381f456288658aee30624499"
format_stamp: "Formatted at 2026-09-28 04:58:02 on dist-test-slave-m39l"
server_key: "b4009147d4c13655c98108aa8c4dcba0"
server_key_iv: "307f9b449e3df8decf6675cc6f95849c"
server_key_version: "encryptionkey@0"
I20260928 04:58:02.146920 19364 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260928 04:58:02.148243 19378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:02.148769 19364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:02.148875 19364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "8338cbf6381f456288658aee30624499"
format_stamp: "Formatted at 2026-09-28 04:58:02 on dist-test-slave-m39l"
server_key: "b4009147d4c13655c98108aa8c4dcba0"
server_key_iv: "307f9b449e3df8decf6675cc6f95849c"
server_key_version: "encryptionkey@0"
I20260928 04:58:02.148950 19364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:02.159729 19364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:02.160056 19364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:02.160176 19364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:02.160385 19364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:58:02.160756 19364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:58:02.160794 19364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:02.160844 19364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:58:02.160876 19364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:02.167320 19364 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43809
I20260928 04:58:02.167416 19491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43809 every 8 connection(s)
I20260928 04:58:02.167775 19364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:58:02.173646 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.168790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41207 (local address 127.0.98.62:45791)
0928 04:58:02.168902 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:02.168904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:02.169158 (+ 254us) server_negotiation.cc:407] Connection header received
0928 04:58:02.169742 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:02.169743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:02.169758 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:02.169809 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:02.170563 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.171071 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.172111 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.172308 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.172409 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:02.173325 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:02.173326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:02.173327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:02.173333 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:02.173357 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:02.173361 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:02.173424 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:02.173523 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 04:58:02.173556 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 04:58:02.173938 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.168745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:41207)
0928 04:58:02.168975 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:02.169064 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:02.169622 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:02.169885 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:02.169889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:02.169898 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:02.170427 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:02.170431 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.171172 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.171173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:02.172013 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:02.172016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.172419 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.172420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:02.172559 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:02.173134 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:58:02.173134 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:02.173173 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:58:02.173200 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:02.173426 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:02.173432 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:02.173436 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:58:02.173528 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 04:58:02.173647 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":100,"threads_started":1}
I20260928 04:58:02.174470 19492 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45791
I20260928 04:58:02.174634 19492 heartbeater.cc:464] Registering TS with master...
I20260928 04:58:02.174857 19492 heartbeater.cc:511] Master 127.0.98.62:45791 requested a full tablet report, sending...
I20260928 04:58:02.175300 19042 ts_manager.cc:194] Registered new tserver with Master: 8338cbf6381f456288658aee30624499 (127.0.98.3:43809)
I20260928 04:58:02.175693 19042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:41207
I20260928 04:58:02.177517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 19364
I20260928 04:58:02.177613 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:58:02.177937 392 external_mini_cluster.cc:1428] Setting key 9e2abb6dfeeb1c7fe3ab2280a667e18a
I20260928 04:58:02.179119 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:58:02.184029 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.180504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.0.98.62:45791)
0928 04:58:02.180603 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:02.180606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:02.180718 (+ 112us) server_negotiation.cc:407] Connection header received
0928 04:58:02.180746 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:02.180747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:02.180757 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:02.180815 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:02.180981 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.181556 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.182318 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.182738 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.182797 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:02.183655 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:02.183657 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:02.183657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:02.183664 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:02.183685 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:02.183689 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:02.183809 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:02.183919 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 04:58:02.183932 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 04:58:02.185824 19042 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:58:02.187670 19042 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:58:02.195428 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.190014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39999 (local address 127.0.0.1:52960)
0928 04:58:02.190584 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:02.190593 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:02.190630 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:02.191237 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:02.191238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:02.191240 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:02.191275 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:02.191276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.192927 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.192928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:02.193927 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:02.193930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.195011 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.195012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:02.195119 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:02.195139 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 04:58:02.195231 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 04:58:02.195317 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":460,"thread_start_us":96,"threads_started":1}
I20260928 04:58:02.195434 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.190014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46327 (local address 127.0.0.1:48722)
0928 04:58:02.190356 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:02.190367 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:02.190423 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:02.190842 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:02.190845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:02.190848 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:02.190921 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:02.190922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.192866 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.192867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:02.194029 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:02.194031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.195010 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.195012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:02.195119 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:02.195139 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 04:58:02.195231 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 04:58:02.195317 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":12,"thread_start_us":121,"threads_started":1}
I20260928 04:58:02.195976 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.190300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48722 (local address 127.0.98.2:46327)
0928 04:58:02.190607 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:02.190612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:02.190627 (+ 15us) server_negotiation.cc:407] Connection header received
0928 04:58:02.190713 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:02.190716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:02.190733 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:02.190808 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:02.191129 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.192757 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.194125 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.194862 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.194955 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:02.195058 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:02.195255 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 04:58:02.195299 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":85,"threads_started":1}
I20260928 04:58:02.196264 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.190014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43809 (local address 127.0.0.1:41036)
0928 04:58:02.190221 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:02.190268 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:02.190338 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:02.191544 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:02.191545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:02.191548 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:02.191597 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:02.191598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.192537 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.192538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:02.194024 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:02.194028 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.196045 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.196046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:02.196112 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:02.196116 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:58:02.196160 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 04:58:02.196205 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
I20260928 04:58:02.196305 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.190323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41036 (local address 127.0.98.3:43809)
0928 04:58:02.191388 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:02.191393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:02.191401 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:58:02.191434 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:02.191435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:02.191448 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:02.191545 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:02.191663 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.192437 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.195115 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.195958 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.196045 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:02.196160 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:02.196184 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 04:58:02.196221 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018,"thread_start_us":96,"threads_started":1}
I20260928 04:58:02.196266 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.190164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52960 (local address 127.0.98.1:39999)
0928 04:58:02.191043 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:02.191049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:02.191063 (+ 14us) server_negotiation.cc:407] Connection header received
0928 04:58:02.191113 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:02.191115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:02.191133 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:02.191235 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:02.191364 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.192859 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.194044 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.194863 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.194955 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:02.195059 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:02.196136 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0928 04:58:02.196189 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":92,"threads_started":1}
I20260928 04:58:02.197079 19164 tablet_service.cc:1467] Processing CreateTablet for tablet 3f08171d52fe46dd96d5406f9a8952ab (DEFAULT_TABLE table=test-table [id=f7acd18f46e948449b3d6ad42537fde1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:58:02.197052 19276 tablet_service.cc:1467] Processing CreateTablet for tablet 3f08171d52fe46dd96d5406f9a8952ab (DEFAULT_TABLE table=test-table [id=f7acd18f46e948449b3d6ad42537fde1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:58:02.197033 19426 tablet_service.cc:1467] Processing CreateTablet for tablet 3f08171d52fe46dd96d5406f9a8952ab (DEFAULT_TABLE table=test-table [id=f7acd18f46e948449b3d6ad42537fde1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:58:02.197367 19164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f08171d52fe46dd96d5406f9a8952ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:02.197383 19276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f08171d52fe46dd96d5406f9a8952ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:02.197383 19426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f08171d52fe46dd96d5406f9a8952ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:02.200430 19506 tablet_bootstrap.cc:492] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9: Bootstrap starting.
I20260928 04:58:02.201005 19507 tablet_bootstrap.cc:492] T 3f08171d52fe46dd96d5406f9a8952ab P 8338cbf6381f456288658aee30624499: Bootstrap starting.
I20260928 04:58:02.201262 19506 tablet_bootstrap.cc:654] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:02.201617 19506 log.cc:824] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:02.201705 19507 tablet_bootstrap.cc:654] T 3f08171d52fe46dd96d5406f9a8952ab P 8338cbf6381f456288658aee30624499: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:02.202005 19507 log.cc:824] T 3f08171d52fe46dd96d5406f9a8952ab P 8338cbf6381f456288658aee30624499: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:02.202386 19508 tablet_bootstrap.cc:492] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774: Bootstrap starting.
I20260928 04:58:02.202785 19506 tablet_bootstrap.cc:492] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9: No bootstrap required, opened a new log
I20260928 04:58:02.202860 19506 ts_tablet_manager.cc:1397] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:58:02.203106 19508 tablet_bootstrap.cc:654] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:02.203197 19507 tablet_bootstrap.cc:492] T 3f08171d52fe46dd96d5406f9a8952ab P 8338cbf6381f456288658aee30624499: No bootstrap required, opened a new log
I20260928 04:58:02.203279 19507 ts_tablet_manager.cc:1397] T 3f08171d52fe46dd96d5406f9a8952ab P 8338cbf6381f456288658aee30624499: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:58:02.203437 19508 log.cc:824] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:02.204342 19508 tablet_bootstrap.cc:492] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774: No bootstrap required, opened a new log
I20260928 04:58:02.204422 19508 ts_tablet_manager.cc:1397] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:58:02.205570 19507 raft_consensus.cc:348] T 3f08171d52fe46dd96d5406f9a8952ab P 8338cbf6381f456288658aee30624499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338cbf6381f456288658aee30624499" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } } peers { permanent_uuid: "eb9520a3161e4a938d4f566112feb5e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39999 } } peers { permanent_uuid: "86024196741445809d7f1e619194e774" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46327 } }
I20260928 04:58:02.205626 19506 raft_consensus.cc:348] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338cbf6381f456288658aee30624499" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } } peers { permanent_uuid: "eb9520a3161e4a938d4f566112feb5e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39999 } } peers { permanent_uuid: "86024196741445809d7f1e619194e774" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46327 } }
I20260928 04:58:02.205755 19507 raft_consensus.cc:374] T 3f08171d52fe46dd96d5406f9a8952ab P 8338cbf6381f456288658aee30624499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:02.205801 19507 raft_consensus.cc:729] T 3f08171d52fe46dd96d5406f9a8952ab P 8338cbf6381f456288658aee30624499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8338cbf6381f456288658aee30624499, State: Initialized, Role: FOLLOWER
I20260928 04:58:02.205801 19506 raft_consensus.cc:374] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:02.205837 19506 raft_consensus.cc:729] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9520a3161e4a938d4f566112feb5e9, State: Initialized, Role: FOLLOWER
I20260928 04:58:02.205960 19507 consensus_queue.cc:260] T 3f08171d52fe46dd96d5406f9a8952ab P 8338cbf6381f456288658aee30624499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338cbf6381f456288658aee30624499" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } } peers { permanent_uuid: "eb9520a3161e4a938d4f566112feb5e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39999 } } peers { permanent_uuid: "86024196741445809d7f1e619194e774" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46327 } }
I20260928 04:58:02.205997 19506 consensus_queue.cc:260] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338cbf6381f456288658aee30624499" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } } peers { permanent_uuid: "eb9520a3161e4a938d4f566112feb5e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39999 } } peers { permanent_uuid: "86024196741445809d7f1e619194e774" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46327 } }
I20260928 04:58:02.206300 19507 ts_tablet_manager.cc:1428] T 3f08171d52fe46dd96d5406f9a8952ab P 8338cbf6381f456288658aee30624499: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:58:02.206326 19506 ts_tablet_manager.cc:1428] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 04:58:02.206452 19492 heartbeater.cc:503] Master 127.0.98.62:45791 was elected leader, sending a full tablet report...
I20260928 04:58:02.206452 19230 heartbeater.cc:503] Master 127.0.98.62:45791 was elected leader, sending a full tablet report...
I20260928 04:58:02.207348 19508 raft_consensus.cc:348] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338cbf6381f456288658aee30624499" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } } peers { permanent_uuid: "eb9520a3161e4a938d4f566112feb5e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39999 } } peers { permanent_uuid: "86024196741445809d7f1e619194e774" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46327 } }
I20260928 04:58:02.207532 19508 raft_consensus.cc:374] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:02.207588 19508 raft_consensus.cc:729] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86024196741445809d7f1e619194e774, State: Initialized, Role: FOLLOWER
I20260928 04:58:02.207727 19508 consensus_queue.cc:260] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338cbf6381f456288658aee30624499" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } } peers { permanent_uuid: "eb9520a3161e4a938d4f566112feb5e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39999 } } peers { permanent_uuid: "86024196741445809d7f1e619194e774" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46327 } }
I20260928 04:58:02.208014 19508 ts_tablet_manager.cc:1428] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 04:58:02.208077 19361 heartbeater.cc:503] Master 127.0.98.62:45791 was elected leader, sending a full tablet report...
W20260928 04:58:02.283411 19362 tablet.cc:2369] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:58:02.294591 19513 raft_consensus.cc:482] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:58:02.294728 19513 raft_consensus.cc:504] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338cbf6381f456288658aee30624499" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } } peers { permanent_uuid: "eb9520a3161e4a938d4f566112feb5e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39999 } } peers { permanent_uuid: "86024196741445809d7f1e619194e774" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46327 } }
I20260928 04:58:02.295195 19513 leader_election.cc:288] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8338cbf6381f456288658aee30624499 (127.0.98.3:43809), 86024196741445809d7f1e619194e774 (127.0.98.2:46327)
I20260928 04:58:02.299297 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.295489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33869 (local address 127.0.98.2:46327)
0928 04:58:02.295531 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:02.295534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:02.295744 (+ 210us) server_negotiation.cc:407] Connection header received
0928 04:58:02.295793 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:02.295793 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:02.295804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:02.295873 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:02.296101 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.296779 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.298244 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.298849 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.298928 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:02.298939 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:02.299181 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 04:58:02.299227 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:58:02.299299 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.295475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41651 (local address 127.0.98.3:43809)
0928 04:58:02.295543 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:02.295545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:02.295556 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:58:02.295619 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:02.295620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:02.295628 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:02.295682 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:02.295883 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.296665 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.298244 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.298852 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.298928 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:02.298939 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:02.299169 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 04:58:02.299215 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 04:58:02.299358 19232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.295401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43809 (local address 127.0.98.1:41651)
0928 04:58:02.295466 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:02.295480 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:02.295530 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:02.295716 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:02.295717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:02.295719 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:02.295800 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:02.295802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.296766 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.296767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:02.298127 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:02.298130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.298958 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.298959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:02.299047 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:02.299063 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 04:58:02.299129 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 04:58:02.299246 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":135}
I20260928 04:58:02.299590 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.295409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46327 (local address 127.0.98.1:33869)
0928 04:58:02.295624 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:02.295632 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:02.295730 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:02.295874 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:02.295875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:02.295876 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:02.295955 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:02.295957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.296859 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.296859 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:02.298127 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:02.298130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.298958 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.298959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:02.299047 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:02.299063 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 04:58:02.299129 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 04:58:02.299240 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":24,"thread_start_us":101,"threads_started":1}
I20260928 04:58:02.299836 19446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f08171d52fe46dd96d5406f9a8952ab" candidate_uuid: "eb9520a3161e4a938d4f566112feb5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8338cbf6381f456288658aee30624499" is_pre_election: true
I20260928 04:58:02.300011 19446 raft_consensus.cc:2454] T 3f08171d52fe46dd96d5406f9a8952ab P 8338cbf6381f456288658aee30624499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb9520a3161e4a938d4f566112feb5e9 in term 0.
I20260928 04:58:02.300006 19315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f08171d52fe46dd96d5406f9a8952ab" candidate_uuid: "eb9520a3161e4a938d4f566112feb5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86024196741445809d7f1e619194e774" is_pre_election: true
I20260928 04:58:02.300185 19315 raft_consensus.cc:2454] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb9520a3161e4a938d4f566112feb5e9 in term 0.
I20260928 04:58:02.300316 19120 leader_election.cc:302] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8338cbf6381f456288658aee30624499, eb9520a3161e4a938d4f566112feb5e9; no voters:
I20260928 04:58:02.300451 19513 raft_consensus.cc:2781] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:58:02.300516 19513 raft_consensus.cc:482] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:58:02.300534 19513 raft_consensus.cc:3037] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:02.301326 19513 raft_consensus.cc:504] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338cbf6381f456288658aee30624499" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } } peers { permanent_uuid: "eb9520a3161e4a938d4f566112feb5e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39999 } } peers { permanent_uuid: "86024196741445809d7f1e619194e774" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46327 } }
I20260928 04:58:02.301489 19513 leader_election.cc:288] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [CANDIDATE]: Term 1 election: Requested vote from peers 8338cbf6381f456288658aee30624499 (127.0.98.3:43809), 86024196741445809d7f1e619194e774 (127.0.98.2:46327)
I20260928 04:58:02.301641 19446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f08171d52fe46dd96d5406f9a8952ab" candidate_uuid: "eb9520a3161e4a938d4f566112feb5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8338cbf6381f456288658aee30624499"
I20260928 04:58:02.301700 19315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f08171d52fe46dd96d5406f9a8952ab" candidate_uuid: "eb9520a3161e4a938d4f566112feb5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86024196741445809d7f1e619194e774"
I20260928 04:58:02.301762 19446 raft_consensus.cc:3037] T 3f08171d52fe46dd96d5406f9a8952ab P 8338cbf6381f456288658aee30624499 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:02.301792 19315 raft_consensus.cc:3037] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:02.302621 19446 raft_consensus.cc:2454] T 3f08171d52fe46dd96d5406f9a8952ab P 8338cbf6381f456288658aee30624499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb9520a3161e4a938d4f566112feb5e9 in term 1.
I20260928 04:58:02.302639 19315 raft_consensus.cc:2454] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb9520a3161e4a938d4f566112feb5e9 in term 1.
I20260928 04:58:02.302816 19118 leader_election.cc:302] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86024196741445809d7f1e619194e774, eb9520a3161e4a938d4f566112feb5e9; no voters:
I20260928 04:58:02.302918 19513 raft_consensus.cc:2781] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:02.303066 19513 raft_consensus.cc:686] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [term 1 LEADER]: Becoming Leader. State: Replica: eb9520a3161e4a938d4f566112feb5e9, State: Running, Role: LEADER
I20260928 04:58:02.303175 19513 consensus_queue.cc:237] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338cbf6381f456288658aee30624499" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } } peers { permanent_uuid: "eb9520a3161e4a938d4f566112feb5e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39999 } } peers { permanent_uuid: "86024196741445809d7f1e619194e774" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46327 } }
I20260928 04:58:02.304100 19042 catalog_manager.cc:5495] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb9520a3161e4a938d4f566112feb5e9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "eb9520a3161e4a938d4f566112feb5e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338cbf6381f456288658aee30624499" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb9520a3161e4a938d4f566112feb5e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86024196741445809d7f1e619194e774" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46327 } health_report { overall_health: UNKNOWN } } }
I20260928 04:58:02.312885 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.310397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52976 (local address 127.0.98.1:39999)
0928 04:58:02.310462 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:02.310464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:02.310474 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:58:02.310540 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:02.310541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:02.310567 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:02.310674 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:58:02.310821 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.311327 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.312113 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:02.312303 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:02.312383 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:02.312529 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:58:02.312633 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:58:02.312739 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:02.312770 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 04:58:02.312782 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:58:02.314772 19446 raft_consensus.cc:1261] T 3f08171d52fe46dd96d5406f9a8952ab P 8338cbf6381f456288658aee30624499 [term 1 FOLLOWER]: Refusing update from remote peer eb9520a3161e4a938d4f566112feb5e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:58:02.314769 19315 raft_consensus.cc:1261] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774 [term 1 FOLLOWER]: Refusing update from remote peer eb9520a3161e4a938d4f566112feb5e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:58:02.315126 19513 consensus_queue.cc:1035] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8338cbf6381f456288658aee30624499" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:58:02.315279 19516 consensus_queue.cc:1035] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86024196741445809d7f1e619194e774" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:58:02.318528 19521 mvcc.cc:205] Tried to move back new op lower bound from 7334180791558512640 to 7334180791514472448. Current Snapshot: MvccSnapshot[applied={T|T < 7334180791558512640}]
I20260928 04:58:02.320572 19041 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:58:02.320693 19041 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:58:02.321455 19041 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d3f066ed70641d88426c3c503a1fb0f: Sending DeleteTablet for 3 replicas of tablet 3f08171d52fe46dd96d5406f9a8952ab
I20260928 04:58:02.321895 19164 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f08171d52fe46dd96d5406f9a8952ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:58:02 UTC) from {username='slave'} at 127.0.0.1:52960
I20260928 04:58:02.321928 19276 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f08171d52fe46dd96d5406f9a8952ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:58:02 UTC) from {username='slave'} at 127.0.0.1:48722
I20260928 04:58:02.322118 19529 tablet_replica.cc:330] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9: stopping tablet replica
I20260928 04:58:02.322151 19426 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f08171d52fe46dd96d5406f9a8952ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:58:02 UTC) from {username='slave'} at 127.0.0.1:41036
I20260928 04:58:02.322211 19529 raft_consensus.cc:2229] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:58:02.322345 19529 raft_consensus.cc:2258] T 3f08171d52fe46dd96d5406f9a8952ab P eb9520a3161e4a938d4f566112feb5e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:58:02.322486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 19102
I20260928 04:58:02.326644 19531 tablet_replica.cc:330] T 3f08171d52fe46dd96d5406f9a8952ab P 8338cbf6381f456288658aee30624499: stopping tablet replica
I20260928 04:58:02.326804 19531 raft_consensus.cc:2229] T 3f08171d52fe46dd96d5406f9a8952ab P 8338cbf6381f456288658aee30624499 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:58:02.326948 19531 raft_consensus.cc:2258] T 3f08171d52fe46dd96d5406f9a8952ab P 8338cbf6381f456288658aee30624499 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:58:02.327158 19530 tablet_replica.cc:330] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774: stopping tablet replica
I20260928 04:58:02.327280 19530 raft_consensus.cc:2229] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:58:02.327399 19530 raft_consensus.cc:2258] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:58:02.327510 19531 ts_tablet_manager.cc:1905] T 3f08171d52fe46dd96d5406f9a8952ab P 8338cbf6381f456288658aee30624499: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:58:02.327901 19530 ts_tablet_manager.cc:1905] T 3f08171d52fe46dd96d5406f9a8952ab P 86024196741445809d7f1e619194e774: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 04:58:02.329177 19028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39999 (error 108)
I20260928 04:58:02.329262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 19233
I20260928 04:58:02.331055 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.330924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39999 (local address 127.0.0.1:52992)
0928 04:58:02.330973 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:02.331021 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 04:58:02.331305 19028 catalog_manager.cc:4584] TS eb9520a3161e4a938d4f566112feb5e9 (127.0.98.1:39999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f08171d52fe46dd96d5406f9a8952ab: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39999: connect: Connection refused (error 111)
I20260928 04:58:02.333834 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.333708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46327 (local address 127.0.0.1:48730)
0928 04:58:02.333767 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:02.333800 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 04:58:02.333829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 19364
I20260928 04:58:02.338866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 19011
I20260928 04:58:02.338966 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:02.338825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43809 (local address 127.0.0.1:41044)
0928 04:58:02.338875 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:02.338904 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:43809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
2026-09-28T04:58:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (752 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:02 dist-test-slave-m39l krb5kdc[19535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:02 dist-test-slave-m39l krb5kdc[19535](info): set up 2 sockets
Sep 28 04:58:02 dist-test-slave-m39l krb5kdc[19535](info): commencing operation
krb5kdc: starting...
W20260928 04:58:03.446828 392 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:03 dist-test-slave-m39l krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:58:03.584440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37343
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:03.669490 19548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:03.669763 19548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:03.669807 19548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:03.671191 19548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:03.671293 19548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:03.671322 19548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:03.671350 19548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:03.671382 19548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:03.673074 19548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:03.673472 19548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:03.673753 19548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:58:03.679351 19548 server_base.cc:1034] running on GCE node
W20260928 04:58:03.679270 19553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:03.679286 19556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:03.679289 19554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:03.679955 19548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:03.680357 19548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:03.681515 19548 hybrid_clock.cc:648] HybridClock initialized: now 1790571483681506 us; error 35 us; skew 500 ppm
Sep 28 04:58:03 dist-test-slave-m39l krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:03.684571 19548 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:58:03.685155 19548 webserver.cc:460] Webserver started at http://127.0.98.62:38755/ using document root <none> and password file <none>
I20260928 04:58:03.685412 19548 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:03.685465 19548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:03.685614 19548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:03.686900 19548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "2f234c9174604bbaaa8d63c350c1efd7"
format_stamp: "Formatted at 2026-09-28 04:58:03 on dist-test-slave-m39l"
server_key: "b882d39e394c94fcc2e4a58b2c7270a0"
server_key_iv: "e432d56ff8fad26069640f059cea2a27"
server_key_version: "encryptionkey@0"
I20260928 04:58:03.687282 19548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "2f234c9174604bbaaa8d63c350c1efd7"
format_stamp: "Formatted at 2026-09-28 04:58:03 on dist-test-slave-m39l"
server_key: "b882d39e394c94fcc2e4a58b2c7270a0"
server_key_iv: "e432d56ff8fad26069640f059cea2a27"
server_key_version: "encryptionkey@0"
I20260928 04:58:03.688773 19548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:58:03.689668 19563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:03.689956 19548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:03.690088 19548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "2f234c9174604bbaaa8d63c350c1efd7"
format_stamp: "Formatted at 2026-09-28 04:58:03 on dist-test-slave-m39l"
server_key: "b882d39e394c94fcc2e4a58b2c7270a0"
server_key_iv: "e432d56ff8fad26069640f059cea2a27"
server_key_version: "encryptionkey@0"
I20260928 04:58:03.690184 19548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:03.705174 19548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:03.707947 19548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:03.708113 19548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:03.712894 19548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37343
I20260928 04:58:03.712929 19615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37343 every 8 connection(s)
I20260928 04:58:03.713377 19548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:03.714329 19616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:03.716902 19616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7: Bootstrap starting.
I20260928 04:58:03.717574 19616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:03.717892 19616 log.cc:824] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:03.718827 19616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7: No bootstrap required, opened a new log
I20260928 04:58:03.720415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 19548
I20260928 04:58:03.720541 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:03.720506 19616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f234c9174604bbaaa8d63c350c1efd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37343 } }
I20260928 04:58:03.720723 19616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:03.720747 392 external_mini_cluster.cc:1428] Setting key 92a8f9b41366bed6e8ce8fa106585a8a
I20260928 04:58:03.720768 19616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f234c9174604bbaaa8d63c350c1efd7, State: Initialized, Role: FOLLOWER
I20260928 04:58:03.720945 19616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f234c9174604bbaaa8d63c350c1efd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37343 } }
I20260928 04:58:03.721051 19616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:03.721091 19616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:03.721138 19616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:03.722376 19616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f234c9174604bbaaa8d63c350c1efd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37343 } }
I20260928 04:58:03.722528 19616 leader_election.cc:302] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f234c9174604bbaaa8d63c350c1efd7; no voters:
I20260928 04:58:03.722724 19616 leader_election.cc:288] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:03.722754 19621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:03.722877 19621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7 [term 1 LEADER]: Becoming Leader. State: Replica: 2f234c9174604bbaaa8d63c350c1efd7, State: Running, Role: LEADER
I20260928 04:58:03.722972 19621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f234c9174604bbaaa8d63c350c1efd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37343 } }
I20260928 04:58:03.723019 19616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:03.723353 19621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f234c9174604bbaaa8d63c350c1efd7. Latest consensus state: current_term: 1 leader_uuid: "2f234c9174604bbaaa8d63c350c1efd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f234c9174604bbaaa8d63c350c1efd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37343 } } }
I20260928 04:58:03.723439 19621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:03.723367 19623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f234c9174604bbaaa8d63c350c1efd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f234c9174604bbaaa8d63c350c1efd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37343 } } }
I20260928 04:58:03.723481 19623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:03.723814 19626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:03.724372 19626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:03.726109 19626 catalog_manager.cc:1347] Generated new cluster ID: 0c5d5444ce4140af9bf1e50ab0d8aec7
I20260928 04:58:03.726172 19626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:58:03 dist-test-slave-m39l krb5kdc[19535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:58:03.733515 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:03.721330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48658 (local address 127.0.98.62:37343)
0928 04:58:03.721592 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:03.721603 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:03.721644 (+ 41us) server_negotiation.cc:407] Connection header received
0928 04:58:03.721795 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:03.721803 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:03.721876 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:03.722083 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:03.722861 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:03.724301 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:03.725325 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:03.725949 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:03.726450 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:03.729286 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:03.729293 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:03.729293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:03.729311 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:03.731593 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:03.732055 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:03.732059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:03.732060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:03.732212 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:03.732418 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:03.732419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:03.732420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:03.732814 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:03.733015 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:03.733116 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 04:58:03.733282 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":127,"threads_started":1}
I20260928 04:58:03.744553 19626 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:03.745350 19626 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:03.754750 19626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7: Generated new TSK 0
I20260928 04:58:03.755021 19626 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:58:03.797634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37343
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:03.886469 19642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:03.886734 19642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:03.886760 19642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:03.887979 19642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:03.888028 19642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:03.888057 19642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:58:03.889878 19642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:03.890147 19642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:03.890344 19642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:03.896147 19650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:03.896147 19648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:03.896147 19647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:03.896451 19642 server_base.cc:1034] running on GCE node
I20260928 04:58:03.896756 19642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:03.897034 19642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:03.898216 19642 hybrid_clock.cc:648] HybridClock initialized: now 1790571483898203 us; error 33 us; skew 500 ppm
Sep 28 04:58:03 dist-test-slave-m39l krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:03.901470 19642 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:58:03.902045 19642 webserver.cc:460] Webserver started at http://127.0.98.1:42811/ using document root <none> and password file <none>
I20260928 04:58:03.902423 19642 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:03.902534 19642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:03.902719 19642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:03.903990 19642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "c1a7820b69c94e8394428834c69a20f1"
format_stamp: "Formatted at 2026-09-28 04:58:03 on dist-test-slave-m39l"
server_key: "79f9d38b7c3e76f963c555a691557e54"
server_key_iv: "9b9a6cd6a7c198d29802b121c88a7df6"
server_key_version: "encryptionkey@0"
I20260928 04:58:03.904423 19642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c1a7820b69c94e8394428834c69a20f1"
format_stamp: "Formatted at 2026-09-28 04:58:03 on dist-test-slave-m39l"
server_key: "79f9d38b7c3e76f963c555a691557e54"
server_key_iv: "9b9a6cd6a7c198d29802b121c88a7df6"
server_key_version: "encryptionkey@0"
I20260928 04:58:03.905979 19642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:58:03.906945 19657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:03.907202 19642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:03.907318 19642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "c1a7820b69c94e8394428834c69a20f1"
format_stamp: "Formatted at 2026-09-28 04:58:03 on dist-test-slave-m39l"
server_key: "79f9d38b7c3e76f963c555a691557e54"
server_key_iv: "9b9a6cd6a7c198d29802b121c88a7df6"
server_key_version: "encryptionkey@0"
I20260928 04:58:03.907398 19642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:03.920255 19642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:03.922744 19642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:03.922914 19642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:03.923254 19642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:58:03.923609 19642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:58:03.923667 19642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:03.923712 19642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:58:03.923744 19642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:03.930140 19642 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42593
I20260928 04:58:03.930272 19770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42593 every 8 connection(s)
I20260928 04:58:03.930632 19642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:58:03.933509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 19642
I20260928 04:58:03.933641 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:58:03.933837 392 external_mini_cluster.cc:1428] Setting key 53d3f9a156145cd349ef7f8cbb7f547e
Sep 28 04:58:03 dist-test-slave-m39l krb5kdc[19535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 04:58:03.941792 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:03.931553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46719 (local address 127.0.98.62:37343)
0928 04:58:03.931607 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:03.931609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:03.931824 (+ 215us) server_negotiation.cc:407] Connection header received
0928 04:58:03.932930 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:03.932931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:03.932950 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:03.933000 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:03.933818 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:03.934621 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:03.935786 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:03.936029 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:03.936113 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:03.938558 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:03.938560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:03.938561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:03.938570 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:03.940129 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:03.940603 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:03.940605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:03.940606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:03.940672 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:03.941033 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:03.941035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:03.941035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:03.941177 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:03.941258 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:03.941584 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 04:58:03.941684 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:58:03.942034 19773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:03.931520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37343 (local address 127.0.98.1:46719)
0928 04:58:03.931678 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:03.931726 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:03.932794 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:03.933084 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:03.933089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:03.933454 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:03.933716 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:03.933723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:03.934720 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:03.934722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:03.935673 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:03.935676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:03.936122 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:03.936124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:03.936280 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:03.936900 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:03.936910 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:03.938422 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:03.940238 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:03.940247 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:03.940250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:03.940518 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:03.940749 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:03.940750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:03.940751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:03.940880 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:03.941262 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:03.941270 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:03.941426 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 04:58:03.941559 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 04:58:03.941750 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260928 04:58:03.942615 19771 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37343
I20260928 04:58:03.942732 19771 heartbeater.cc:464] Registering TS with master...
I20260928 04:58:03.942945 19771 heartbeater.cc:511] Master 127.0.98.62:37343 requested a full tablet report, sending...
I20260928 04:58:03.943645 19580 ts_manager.cc:194] Registered new tserver with Master: c1a7820b69c94e8394428834c69a20f1 (127.0.98.1:42593)
I20260928 04:58:03.944319 19580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:58:03.963744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37343
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:04.049978 19776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:04.050253 19776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:04.050299 19776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:04.051684 19776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:04.051769 19776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:04.051817 19776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:58:04.053464 19776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:04.053772 19776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:04.054021 19776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:04.059689 19782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:04.059773 19776 server_base.cc:1034] running on GCE node
W20260928 04:58:04.059689 19781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:04.059713 19784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:04.060302 19776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:04.060595 19776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:04.061751 19776 hybrid_clock.cc:648] HybridClock initialized: now 1790571484061732 us; error 37 us; skew 500 ppm
Sep 28 04:58:04 dist-test-slave-m39l krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:04.064626 19776 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:58:04.065240 19776 webserver.cc:460] Webserver started at http://127.0.98.2:41195/ using document root <none> and password file <none>
I20260928 04:58:04.065533 19776 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:04.065589 19776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:04.065788 19776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:04.067203 19776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "4c231e82339c40a7afac09189263c531"
format_stamp: "Formatted at 2026-09-28 04:58:04 on dist-test-slave-m39l"
server_key: "dd793acb5f81812c62896271c3d2c68d"
server_key_iv: "51b7a0c54e6a1031ebeb38a16b211423"
server_key_version: "encryptionkey@0"
I20260928 04:58:04.067715 19776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4c231e82339c40a7afac09189263c531"
format_stamp: "Formatted at 2026-09-28 04:58:04 on dist-test-slave-m39l"
server_key: "dd793acb5f81812c62896271c3d2c68d"
server_key_iv: "51b7a0c54e6a1031ebeb38a16b211423"
server_key_version: "encryptionkey@0"
I20260928 04:58:04.069523 19776 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 04:58:04.070570 19791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:04.070777 19776 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:04.070883 19776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "4c231e82339c40a7afac09189263c531"
format_stamp: "Formatted at 2026-09-28 04:58:04 on dist-test-slave-m39l"
server_key: "dd793acb5f81812c62896271c3d2c68d"
server_key_iv: "51b7a0c54e6a1031ebeb38a16b211423"
server_key_version: "encryptionkey@0"
I20260928 04:58:04.070973 19776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:04.085484 19776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:04.088539 19776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:04.088706 19776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:04.088951 19776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:58:04.089306 19776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:58:04.089357 19776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:04.089393 19776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:58:04.089421 19776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:04.095920 19776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35167
I20260928 04:58:04.095963 19904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35167 every 8 connection(s)
I20260928 04:58:04.096326 19776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:58:04.099170 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 19776
I20260928 04:58:04.099296 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:58:04.099485 392 external_mini_cluster.cc:1428] Setting key f75310e175abab0648a3485be9f8eca7
Sep 28 04:58:04 dist-test-slave-m39l krb5kdc[19535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:58:04.107962 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.097102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40995 (local address 127.0.98.62:37343)
0928 04:58:04.097152 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:04.097158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:04.097354 (+ 196us) server_negotiation.cc:407] Connection header received
0928 04:58:04.098300 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:04.098301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:04.098312 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:04.098378 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:04.099142 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.099931 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.101340 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.101678 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.101764 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.104690 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:04.104692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:04.104693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:04.104707 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:04.106330 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:04.106878 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:04.106884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:04.106885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:04.106939 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:04.107275 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:04.107276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:04.107276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:04.107401 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:04.107473 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:04.107767 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 04:58:04.107865 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:58:04.108173 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.097035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37343 (local address 127.0.98.2:40995)
0928 04:58:04.097216 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:04.097263 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:04.098206 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:04.098448 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:04.098451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:04.098829 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:04.099035 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:04.099041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.100103 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.100104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:04.101249 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:04.101253 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.101783 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.101784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:04.101960 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.102728 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:04.102740 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:04.104569 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:04.106456 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:04.106468 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:04.106526 (+ 58us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:04.106772 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:04.107080 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:04.107081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:04.107081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:04.107199 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:04.107464 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:04.107479 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:04.107630 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 04:58:04.107734 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 04:58:04.107907 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 04:58:04.108992 19905 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37343
I20260928 04:58:04.109124 19905 heartbeater.cc:464] Registering TS with master...
I20260928 04:58:04.109313 19905 heartbeater.cc:511] Master 127.0.98.62:37343 requested a full tablet report, sending...
I20260928 04:58:04.109733 19580 ts_manager.cc:194] Registered new tserver with Master: 4c231e82339c40a7afac09189263c531 (127.0.98.2:35167)
I20260928 04:58:04.110194 19580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:58:04.127364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37343
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:04.213675 19910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:04.213913 19910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:04.213943 19910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:04.215263 19910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:04.215327 19910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:04.215359 19910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:58:04.217056 19910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:04.217367 19910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:04.217653 19910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:04.223577 19915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:04.223548 19918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:04.223548 19916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:04.223959 19910 server_base.cc:1034] running on GCE node
I20260928 04:58:04.224234 19910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:04.224506 19910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:04.225708 19910 hybrid_clock.cc:648] HybridClock initialized: now 1790571484225686 us; error 54 us; skew 500 ppm
Sep 28 04:58:04 dist-test-slave-m39l krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:04.228741 19910 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:58:04.229310 19910 webserver.cc:460] Webserver started at http://127.0.98.3:40049/ using document root <none> and password file <none>
I20260928 04:58:04.229580 19910 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:04.229658 19910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:04.229812 19910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:04.231076 19910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "9b29639e9f114c34972bc5d0c5d9460c"
format_stamp: "Formatted at 2026-09-28 04:58:04 on dist-test-slave-m39l"
server_key: "3614beb1f133919b9771fd4915ff1871"
server_key_iv: "345314948e5634d9f3ca7ed3f7c77a82"
server_key_version: "encryptionkey@0"
I20260928 04:58:04.231477 19910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9b29639e9f114c34972bc5d0c5d9460c"
format_stamp: "Formatted at 2026-09-28 04:58:04 on dist-test-slave-m39l"
server_key: "3614beb1f133919b9771fd4915ff1871"
server_key_iv: "345314948e5634d9f3ca7ed3f7c77a82"
server_key_version: "encryptionkey@0"
I20260928 04:58:04.232975 19910 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:58:04.233841 19925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:04.234025 19910 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:04.234135 19910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "9b29639e9f114c34972bc5d0c5d9460c"
format_stamp: "Formatted at 2026-09-28 04:58:04 on dist-test-slave-m39l"
server_key: "3614beb1f133919b9771fd4915ff1871"
server_key_iv: "345314948e5634d9f3ca7ed3f7c77a82"
server_key_version: "encryptionkey@0"
I20260928 04:58:04.234220 19910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:04.261417 19910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:04.263979 19910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:04.264165 19910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:04.264428 19910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:58:04.264776 19910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:58:04.264832 19910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:04.264874 19910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:58:04.264905 19910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:04.272451 19910 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37645
I20260928 04:58:04.272500 20038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37645 every 8 connection(s)
I20260928 04:58:04.272886 19910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:58:04.273315 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 19910
I20260928 04:58:04.273422 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:58:04.273608 392 external_mini_cluster.cc:1428] Setting key 1c3e949bdb19bbb1bd5bd7633fd5325b
Sep 28 04:58:04 dist-test-slave-m39l krb5kdc[19535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:58:04.283622 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.274132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43469 (local address 127.0.98.62:37343)
0928 04:58:04.274176 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:04.274178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:04.274216 (+ 38us) server_negotiation.cc:407] Connection header received
0928 04:58:04.275430 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:04.275431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:04.275447 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:04.275511 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:04.276176 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.276706 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.277789 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.277987 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.278044 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.280451 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:04.280452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:04.280453 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:04.280461 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:04.282102 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:04.282631 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:04.282632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:04.282633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:04.282695 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:04.282979 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:04.282981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:04.282981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:04.283124 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:04.283183 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:04.283474 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 04:58:04.283554 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:58:04.283799 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.273782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37343 (local address 127.0.98.3:43469)
0928 04:58:04.274076 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:04.274136 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:04.275310 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:04.275593 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:04.275597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:04.275880 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:04.276079 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:04.276084 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.276794 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.276795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:04.277706 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:04.277709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.278063 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.278063 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:04.278225 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.278826 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:04.278834 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:04.280324 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:04.282191 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:04.282198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:04.282200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:04.282437 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:04.282761 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:04.282762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:04.282762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:04.282880 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:04.283185 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:04.283189 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:04.283325 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 04:58:04.283409 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 04:58:04.283546 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":128,"threads_started":1}
I20260928 04:58:04.284355 20039 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37343
I20260928 04:58:04.284468 20039 heartbeater.cc:464] Registering TS with master...
I20260928 04:58:04.284682 20039 heartbeater.cc:511] Master 127.0.98.62:37343 requested a full tablet report, sending...
I20260928 04:58:04.285259 19580 ts_manager.cc:194] Registered new tserver with Master: 9b29639e9f114c34972bc5d0c5d9460c (127.0.98.3:37645)
I20260928 04:58:04.285686 19580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43469
I20260928 04:58:04.285696 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:58:04.294252 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.287322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.0.98.62:37343)
0928 04:58:04.287412 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:04.287415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:04.287514 (+ 99us) server_negotiation.cc:407] Connection header received
0928 04:58:04.287643 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:04.287644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:04.287654 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:04.287712 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:04.288343 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.288849 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.289594 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.289788 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.289865 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.291238 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:04.291240 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:04.291240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:04.291249 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:04.292790 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:04.293251 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:04.293253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:04.293253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:04.293319 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:04.293560 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:04.293561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:04.293562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:04.293732 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:04.293844 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:04.294030 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 04:58:04.294127 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 04:58:04.296120 19580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:58:04.298058 19580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:58:04.305305 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.300935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34806 (local address 127.0.98.1:42593)
0928 04:58:04.301135 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:04.301141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:04.301151 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:58:04.301191 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:04.301193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:04.301209 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:04.301292 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:04.301524 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.302716 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.304251 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.304955 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.305013 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.305071 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:04.305170 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 04:58:04.305208 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
I20260928 04:58:04.305280 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.300579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35167 (local address 127.0.0.1:54844)
0928 04:58:04.301563 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:04.301573 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:04.301625 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:04.301804 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:04.301805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:04.301807 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:04.301849 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:04.301851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.303023 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.303024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:04.303957 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:04.303960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.304881 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.304883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:04.304989 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.305008 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 04:58:04.305085 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 04:58:04.305137 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"mutex_wait_us":576,"thread_start_us":193,"threads_started":1}
I20260928 04:58:04.305644 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.300725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.0.98.3:37645)
0928 04:58:04.300889 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:04.300895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:04.300908 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:58:04.301027 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:04.301029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:04.301048 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:04.301141 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:04.301438 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.302450 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.304271 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.304814 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.305323 (+ 509us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.305473 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:04.305509 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 04:58:04.305560 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260928 04:58:04.305922 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.300731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54844 (local address 127.0.98.2:35167)
0928 04:58:04.301449 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:04.301453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:04.301625 (+ 172us) server_negotiation.cc:407] Connection header received
0928 04:58:04.301698 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:04.301700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:04.301716 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:04.301790 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:04.301918 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.302926 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.304045 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.304783 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.304863 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.304926 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:04.305816 (+ 890us) server_negotiation.cc:299] Negotiation successful
0928 04:58:04.305856 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":95,"threads_started":1}
I20260928 04:58:04.305296 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.300579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42593 (local address 127.0.0.1:34806)
0928 04:58:04.300912 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:04.300924 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:04.300985 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:04.301390 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:04.301391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:04.301394 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:04.301441 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:04.301454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.302818 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.302818 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:04.304167 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:04.304169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.305027 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.305028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:04.305096 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.305100 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:58:04.305155 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 04:58:04.305208 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":6,"thread_start_us":106,"threads_started":1}
I20260928 04:58:04.305280 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.300578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37645 (local address 127.0.0.1:50874)
0928 04:58:04.300788 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:04.300838 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:04.300911 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:04.301170 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:04.301173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:04.301176 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:04.301338 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:04.301341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.302629 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.302631 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:04.304154 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:04.304158 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.304939 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.304940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:04.305010 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.305013 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:58:04.305063 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 04:58:04.305137 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":39,"thread_start_us":77,"threads_started":1}
I20260928 04:58:04.306986 19973 tablet_service.cc:1467] Processing CreateTablet for tablet 101f18ac3a4c41f5ac10ffe9c3cf5d78 (DEFAULT_TABLE table=test-table [id=ffc444d6cf444217a84a8ad515b43a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:58:04.306965 19705 tablet_service.cc:1467] Processing CreateTablet for tablet 101f18ac3a4c41f5ac10ffe9c3cf5d78 (DEFAULT_TABLE table=test-table [id=ffc444d6cf444217a84a8ad515b43a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:58:04.307065 19839 tablet_service.cc:1467] Processing CreateTablet for tablet 101f18ac3a4c41f5ac10ffe9c3cf5d78 (DEFAULT_TABLE table=test-table [id=ffc444d6cf444217a84a8ad515b43a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:58:04.307358 19973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101f18ac3a4c41f5ac10ffe9c3cf5d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:04.307356 19839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101f18ac3a4c41f5ac10ffe9c3cf5d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:04.307353 19705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101f18ac3a4c41f5ac10ffe9c3cf5d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:04.310137 20056 tablet_bootstrap.cc:492] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c: Bootstrap starting.
I20260928 04:58:04.310977 20056 tablet_bootstrap.cc:654] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:04.311303 20057 tablet_bootstrap.cc:492] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1: Bootstrap starting.
I20260928 04:58:04.311638 20056 log.cc:824] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:04.312040 20057 tablet_bootstrap.cc:654] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:04.312563 20057 log.cc:824] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:04.312578 20056 tablet_bootstrap.cc:492] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c: No bootstrap required, opened a new log
I20260928 04:58:04.312668 20056 ts_tablet_manager.cc:1397] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 04:58:04.312803 20058 tablet_bootstrap.cc:492] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531: Bootstrap starting.
I20260928 04:58:04.313376 20058 tablet_bootstrap.cc:654] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:04.313383 20057 tablet_bootstrap.cc:492] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1: No bootstrap required, opened a new log
I20260928 04:58:04.313570 20057 ts_tablet_manager.cc:1397] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:58:04.313814 20058 log.cc:824] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:04.314433 20058 tablet_bootstrap.cc:492] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531: No bootstrap required, opened a new log
I20260928 04:58:04.314538 20058 ts_tablet_manager.cc:1397] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 04:58:04.315069 20056 raft_consensus.cc:348] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b29639e9f114c34972bc5d0c5d9460c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37645 } } peers { permanent_uuid: "c1a7820b69c94e8394428834c69a20f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42593 } } peers { permanent_uuid: "4c231e82339c40a7afac09189263c531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35167 } }
I20260928 04:58:04.315289 20056 raft_consensus.cc:374] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:04.315341 20056 raft_consensus.cc:729] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b29639e9f114c34972bc5d0c5d9460c, State: Initialized, Role: FOLLOWER
I20260928 04:58:04.315474 20056 consensus_queue.cc:260] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b29639e9f114c34972bc5d0c5d9460c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37645 } } peers { permanent_uuid: "c1a7820b69c94e8394428834c69a20f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42593 } } peers { permanent_uuid: "4c231e82339c40a7afac09189263c531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35167 } }
I20260928 04:58:04.315810 20056 ts_tablet_manager.cc:1428] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:58:04.315835 20039 heartbeater.cc:503] Master 127.0.98.62:37343 was elected leader, sending a full tablet report...
I20260928 04:58:04.316210 20058 raft_consensus.cc:348] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b29639e9f114c34972bc5d0c5d9460c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37645 } } peers { permanent_uuid: "c1a7820b69c94e8394428834c69a20f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42593 } } peers { permanent_uuid: "4c231e82339c40a7afac09189263c531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35167 } }
I20260928 04:58:04.316390 20058 raft_consensus.cc:374] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:04.316431 20058 raft_consensus.cc:729] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c231e82339c40a7afac09189263c531, State: Initialized, Role: FOLLOWER
I20260928 04:58:04.316538 20058 consensus_queue.cc:260] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b29639e9f114c34972bc5d0c5d9460c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37645 } } peers { permanent_uuid: "c1a7820b69c94e8394428834c69a20f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42593 } } peers { permanent_uuid: "4c231e82339c40a7afac09189263c531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35167 } }
I20260928 04:58:04.316675 20057 raft_consensus.cc:348] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b29639e9f114c34972bc5d0c5d9460c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37645 } } peers { permanent_uuid: "c1a7820b69c94e8394428834c69a20f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42593 } } peers { permanent_uuid: "4c231e82339c40a7afac09189263c531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35167 } }
I20260928 04:58:04.316829 20058 ts_tablet_manager.cc:1428] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:58:04.316869 19905 heartbeater.cc:503] Master 127.0.98.62:37343 was elected leader, sending a full tablet report...
I20260928 04:58:04.316869 20057 raft_consensus.cc:374] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:04.316916 20057 raft_consensus.cc:729] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1a7820b69c94e8394428834c69a20f1, State: Initialized, Role: FOLLOWER
I20260928 04:58:04.317035 20057 consensus_queue.cc:260] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b29639e9f114c34972bc5d0c5d9460c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37645 } } peers { permanent_uuid: "c1a7820b69c94e8394428834c69a20f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42593 } } peers { permanent_uuid: "4c231e82339c40a7afac09189263c531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35167 } }
I20260928 04:58:04.317373 20057 ts_tablet_manager.cc:1428] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 04:58:04.317373 19771 heartbeater.cc:503] Master 127.0.98.62:37343 was elected leader, sending a full tablet report...
W20260928 04:58:04.347021 19906 tablet.cc:2369] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:58:04.431461 19772 tablet.cc:2369] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:58:04.469771 20064 raft_consensus.cc:482] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:58:04.469918 20064 raft_consensus.cc:504] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b29639e9f114c34972bc5d0c5d9460c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37645 } } peers { permanent_uuid: "c1a7820b69c94e8394428834c69a20f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42593 } } peers { permanent_uuid: "4c231e82339c40a7afac09189263c531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35167 } }
I20260928 04:58:04.470398 20064 leader_election.cc:288] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b29639e9f114c34972bc5d0c5d9460c (127.0.98.3:37645), 4c231e82339c40a7afac09189263c531 (127.0.98.2:35167)
I20260928 04:58:04.473613 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.470678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37645 (local address 127.0.98.1:35497)
0928 04:58:04.470849 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:04.470858 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:04.470988 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:04.471148 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:04.471149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:04.471151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:04.471244 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:04.471245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.471960 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.471961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:04.472793 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:04.472796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.473344 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.473345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:04.473416 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.473420 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:58:04.473474 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 04:58:04.473514 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260928 04:58:04.473621 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.470721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35497 (local address 127.0.98.3:37645)
0928 04:58:04.470786 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:04.470788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:04.470916 (+ 128us) server_negotiation.cc:407] Connection header received
0928 04:58:04.471061 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:04.471062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:04.471072 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:04.471143 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:04.471315 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.471805 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.472886 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.473257 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.473341 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.473351 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:04.473484 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 04:58:04.473534 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:58:04.473737 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.470732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37137 (local address 127.0.98.2:35167)
0928 04:58:04.470799 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:04.470802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:04.470961 (+ 159us) server_negotiation.cc:407] Connection header received
0928 04:58:04.471042 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:04.471043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:04.471063 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:04.471132 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:04.471301 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.471821 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.472901 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.473312 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.473397 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.473409 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:04.473601 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 04:58:04.473662 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 04:58:04.473755 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.470680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35167 (local address 127.0.98.1:37137)
0928 04:58:04.470864 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:04.470874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:04.470964 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:04.471137 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:04.471138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:04.471140 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:04.471220 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:04.471222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.471921 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.471922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:04.472796 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:04.472797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.473432 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.473434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:04.473520 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.473525 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:58:04.473584 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 04:58:04.473626 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":19,"thread_start_us":70,"threads_started":1}
I20260928 04:58:04.474023 19993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101f18ac3a4c41f5ac10ffe9c3cf5d78" candidate_uuid: "c1a7820b69c94e8394428834c69a20f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b29639e9f114c34972bc5d0c5d9460c" is_pre_election: true
I20260928 04:58:04.474184 19993 raft_consensus.cc:2454] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1a7820b69c94e8394428834c69a20f1 in term 0.
I20260928 04:58:04.474140 19859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101f18ac3a4c41f5ac10ffe9c3cf5d78" candidate_uuid: "c1a7820b69c94e8394428834c69a20f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c231e82339c40a7afac09189263c531" is_pre_election: true
I20260928 04:58:04.474306 19859 raft_consensus.cc:2454] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1a7820b69c94e8394428834c69a20f1 in term 0.
I20260928 04:58:04.474429 19660 leader_election.cc:302] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b29639e9f114c34972bc5d0c5d9460c, c1a7820b69c94e8394428834c69a20f1; no voters:
I20260928 04:58:04.474603 20064 raft_consensus.cc:2781] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:58:04.474658 20064 raft_consensus.cc:482] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:58:04.474675 20064 raft_consensus.cc:3037] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:04.475387 20064 raft_consensus.cc:504] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b29639e9f114c34972bc5d0c5d9460c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37645 } } peers { permanent_uuid: "c1a7820b69c94e8394428834c69a20f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42593 } } peers { permanent_uuid: "4c231e82339c40a7afac09189263c531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35167 } }
I20260928 04:58:04.475642 20064 leader_election.cc:288] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [CANDIDATE]: Term 1 election: Requested vote from peers 9b29639e9f114c34972bc5d0c5d9460c (127.0.98.3:37645), 4c231e82339c40a7afac09189263c531 (127.0.98.2:35167)
I20260928 04:58:04.475685 19993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101f18ac3a4c41f5ac10ffe9c3cf5d78" candidate_uuid: "c1a7820b69c94e8394428834c69a20f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b29639e9f114c34972bc5d0c5d9460c"
I20260928 04:58:04.475785 19993 raft_consensus.cc:3037] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:04.475898 19859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101f18ac3a4c41f5ac10ffe9c3cf5d78" candidate_uuid: "c1a7820b69c94e8394428834c69a20f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c231e82339c40a7afac09189263c531"
I20260928 04:58:04.476089 19859 raft_consensus.cc:3037] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:04.476469 19993 raft_consensus.cc:2454] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1a7820b69c94e8394428834c69a20f1 in term 1.
I20260928 04:58:04.476632 19660 leader_election.cc:302] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b29639e9f114c34972bc5d0c5d9460c, c1a7820b69c94e8394428834c69a20f1; no voters:
I20260928 04:58:04.476727 20064 raft_consensus.cc:2781] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:04.476889 20064 raft_consensus.cc:686] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [term 1 LEADER]: Becoming Leader. State: Replica: c1a7820b69c94e8394428834c69a20f1, State: Running, Role: LEADER
I20260928 04:58:04.476997 20064 consensus_queue.cc:237] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b29639e9f114c34972bc5d0c5d9460c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37645 } } peers { permanent_uuid: "c1a7820b69c94e8394428834c69a20f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42593 } } peers { permanent_uuid: "4c231e82339c40a7afac09189263c531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35167 } }
I20260928 04:58:04.477104 19859 raft_consensus.cc:2454] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1a7820b69c94e8394428834c69a20f1 in term 1.
I20260928 04:58:04.477872 19579 catalog_manager.cc:5495] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1a7820b69c94e8394428834c69a20f1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c1a7820b69c94e8394428834c69a20f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b29639e9f114c34972bc5d0c5d9460c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1a7820b69c94e8394428834c69a20f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c231e82339c40a7afac09189263c531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35167 } health_report { overall_health: UNKNOWN } } }
I20260928 04:58:04.488680 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.486163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34822 (local address 127.0.98.1:42593)
0928 04:58:04.486222 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:04.486224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:04.486257 (+ 33us) server_negotiation.cc:407] Connection header received
0928 04:58:04.486336 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:04.486337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:04.486358 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:04.486454 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:58:04.486662 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.487167 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.487991 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.488192 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.488250 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.488379 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:58:04.488485 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:58:04.488545 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:04.488566 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 04:58:04.488584 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 04:58:04.490739 19993 raft_consensus.cc:1261] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c [term 1 FOLLOWER]: Refusing update from remote peer c1a7820b69c94e8394428834c69a20f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:58:04.490787 19859 raft_consensus.cc:1261] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531 [term 1 FOLLOWER]: Refusing update from remote peer c1a7820b69c94e8394428834c69a20f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:58:04.491191 20067 consensus_queue.cc:1035] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c231e82339c40a7afac09189263c531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:58:04.491329 20064 consensus_queue.cc:1035] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b29639e9f114c34972bc5d0c5d9460c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:58:04.494382 20072 mvcc.cc:205] Tried to move back new op lower bound from 7334180800470884352 to 7334180800418480128. Current Snapshot: MvccSnapshot[applied={T|T < 7334180800470884352 or (T in {7334180800470884352})}]
I20260928 04:58:04.495119 20071 mvcc.cc:205] Tried to move back new op lower bound from 7334180800470884352 to 7334180800418480128. Current Snapshot: MvccSnapshot[applied={T|T < 7334180800470884352 or (T in {7334180800470884352})}]
I20260928 04:58:04.496888 19578 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:58:04.497004 19578 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:58:04.497816 19578 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f234c9174604bbaaa8d63c350c1efd7: Sending DeleteTablet for 3 replicas of tablet 101f18ac3a4c41f5ac10ffe9c3cf5d78
I20260928 04:58:04.498201 19839 tablet_service.cc:1514] Processing DeleteTablet for tablet 101f18ac3a4c41f5ac10ffe9c3cf5d78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:58:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54844
I20260928 04:58:04.498189 19705 tablet_service.cc:1514] Processing DeleteTablet for tablet 101f18ac3a4c41f5ac10ffe9c3cf5d78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:58:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34806
I20260928 04:58:04.498440 20080 tablet_replica.cc:330] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1: stopping tablet replica
I20260928 04:58:04.498440 19973 tablet_service.cc:1514] Processing DeleteTablet for tablet 101f18ac3a4c41f5ac10ffe9c3cf5d78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:58:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50874
I20260928 04:58:04.498607 20080 raft_consensus.cc:2229] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:58:04.498752 20080 raft_consensus.cc:2258] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P c1a7820b69c94e8394428834c69a20f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:58:04.498919 20081 tablet_replica.cc:330] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531: stopping tablet replica
I20260928 04:58:04.499017 20081 raft_consensus.cc:2229] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:58:04.499049 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 19642
I20260928 04:58:04.499890 20081 raft_consensus.cc:2258] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:58:04.500398 20081 ts_tablet_manager.cc:1905] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:58:04.502259 20081 ts_tablet_manager.cc:1918] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:58:04.502333 20081 log.cc:1196] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/101f18ac3a4c41f5ac10ffe9c3cf5d78
I20260928 04:58:04.502678 20081 ts_tablet_manager.cc:1939] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 4c231e82339c40a7afac09189263c531: Deleting consensus metadata
I20260928 04:58:04.503194 19564 catalog_manager.cc:4841] TS 4c231e82339c40a7afac09189263c531 (127.0.98.2:35167): tablet 101f18ac3a4c41f5ac10ffe9c3cf5d78 (table test-table [id=ffc444d6cf444217a84a8ad515b43a94]) successfully deleted
W20260928 04:58:04.504169 19565 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42593 (error 108)
I20260928 04:58:04.504392 20082 tablet_replica.cc:330] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c: stopping tablet replica
I20260928 04:58:04.504550 20082 raft_consensus.cc:2229] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:58:04.504679 20082 raft_consensus.cc:2258] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:58:04.504846 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.504698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42593 (local address 127.0.0.1:34838)
0928 04:58:04.504759 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:04.504805 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 04:58:04.504941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 19776
W20260928 04:58:04.504951 19565 catalog_manager.cc:4584] TS c1a7820b69c94e8394428834c69a20f1 (127.0.98.1:42593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 101f18ac3a4c41f5ac10ffe9c3cf5d78: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42593: connect: Connection refused (error 111)
I20260928 04:58:04.505388 20082 ts_tablet_manager.cc:1905] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:58:04.507625 20082 ts_tablet_manager.cc:1918] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:58:04.507719 20082 log.cc:1196] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/101f18ac3a4c41f5ac10ffe9c3cf5d78
I20260928 04:58:04.508049 20082 ts_tablet_manager.cc:1939] T 101f18ac3a4c41f5ac10ffe9c3cf5d78 P 9b29639e9f114c34972bc5d0c5d9460c: Deleting consensus metadata
I20260928 04:58:04.508857 19566 catalog_manager.cc:4841] TS 9b29639e9f114c34972bc5d0c5d9460c (127.0.98.3:37645): tablet 101f18ac3a4c41f5ac10ffe9c3cf5d78 (table test-table [id=ffc444d6cf444217a84a8ad515b43a94]) successfully deleted
I20260928 04:58:04.511844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 19910
I20260928 04:58:04.517441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 19548
2026-09-28T04:58:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2179 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T04:58:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:04Z Disabled control of system clock
I20260928 04:58:04.547084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44043
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:58:04.637558 20087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:04.637827 20087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:04.637850 20087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:04.639580 20087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:04.639657 20087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:04.639670 20087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:04.639685 20087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:04.639698 20087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:04.641355 20087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:04.641687 20087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:04.642048 20087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:04.647495 20093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:04.647495 20092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:04.647650 20095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:04.647743 20087 server_base.cc:1034] running on GCE node
I20260928 04:58:04.648061 20087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:04.648365 20087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:04.649577 20087 hybrid_clock.cc:648] HybridClock initialized: now 1790571484649551 us; error 39 us; skew 500 ppm
I20260928 04:58:04.651243 20087 webserver.cc:460] Webserver started at http://127.0.98.62:34635/ using document root <none> and password file <none>
I20260928 04:58:04.651549 20087 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:04.651609 20087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:04.651752 20087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:04.652931 20087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "4d02e0b3345f48d099cb7460c22e35eb"
format_stamp: "Formatted at 2026-09-28 04:58:04 on dist-test-slave-m39l"
server_key: "0b1a184c8e0c9783532193c59ab17d84"
server_key_iv: "b1f763e89734cb34dae745a4ecfb4971"
server_key_version: "encryptionkey@0"
I20260928 04:58:04.653316 20087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d02e0b3345f48d099cb7460c22e35eb"
format_stamp: "Formatted at 2026-09-28 04:58:04 on dist-test-slave-m39l"
server_key: "0b1a184c8e0c9783532193c59ab17d84"
server_key_iv: "b1f763e89734cb34dae745a4ecfb4971"
server_key_version: "encryptionkey@0"
I20260928 04:58:04.654808 20087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 04:58:04.655632 20101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:04.655824 20087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:04.655931 20087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "4d02e0b3345f48d099cb7460c22e35eb"
format_stamp: "Formatted at 2026-09-28 04:58:04 on dist-test-slave-m39l"
server_key: "0b1a184c8e0c9783532193c59ab17d84"
server_key_iv: "b1f763e89734cb34dae745a4ecfb4971"
server_key_version: "encryptionkey@0"
I20260928 04:58:04.656011 20087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:04.673720 20087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:04.674098 20087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:04.674264 20087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:04.678306 20087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44043
I20260928 04:58:04.678367 20153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44043 every 8 connection(s)
I20260928 04:58:04.678761 20087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:04.679773 20154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:04.682080 20154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb: Bootstrap starting.
I20260928 04:58:04.682471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 20087
I20260928 04:58:04.682617 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:04.682816 392 external_mini_cluster.cc:1428] Setting key 21303266a426bda9790bb9efb09b57ae
I20260928 04:58:04.682868 20154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:04.683218 20154 log.cc:824] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:04.684077 20154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb: No bootstrap required, opened a new log
I20260928 04:58:04.686697 20154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02e0b3345f48d099cb7460c22e35eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44043 } }
I20260928 04:58:04.686858 20154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:04.686900 20154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d02e0b3345f48d099cb7460c22e35eb, State: Initialized, Role: FOLLOWER
I20260928 04:58:04.687039 20154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02e0b3345f48d099cb7460c22e35eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44043 } }
I20260928 04:58:04.687114 20154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:04.687158 20154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:04.687207 20154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:04.687896 20154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02e0b3345f48d099cb7460c22e35eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44043 } }
I20260928 04:58:04.688005 20154 leader_election.cc:302] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d02e0b3345f48d099cb7460c22e35eb; no voters:
I20260928 04:58:04.688197 20154 leader_election.cc:288] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:04.688277 20159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:04.688447 20159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb [term 1 LEADER]: Becoming Leader. State: Replica: 4d02e0b3345f48d099cb7460c22e35eb, State: Running, Role: LEADER
I20260928 04:58:04.688485 20154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:04.688617 20159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02e0b3345f48d099cb7460c22e35eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44043 } }
I20260928 04:58:04.688963 20159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d02e0b3345f48d099cb7460c22e35eb. Latest consensus state: current_term: 1 leader_uuid: "4d02e0b3345f48d099cb7460c22e35eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02e0b3345f48d099cb7460c22e35eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44043 } } }
I20260928 04:58:04.689046 20159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:04.689299 20160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d02e0b3345f48d099cb7460c22e35eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02e0b3345f48d099cb7460c22e35eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44043 } } }
I20260928 04:58:04.689390 20160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:04.689664 20164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:04.690228 20164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:04.690474 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.683411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39234 (local address 127.0.98.62:44043)
0928 04:58:04.683693 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:04.683700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:04.683749 (+ 49us) server_negotiation.cc:407] Connection header received
0928 04:58:04.684365 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:04.684375 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:04.684456 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:04.684618 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:04.684897 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.686518 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.687684 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.688358 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.688888 (+ 530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.689454 (+ 566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:04.689461 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:04.689461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:04.689679 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:04.689725 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:04.689743 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:04.689813 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:04.690054 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 04:58:04.690142 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
I20260928 04:58:04.692297 20164 catalog_manager.cc:1347] Generated new cluster ID: 28f3448b9b2448e5a26361b851f0358a
I20260928 04:58:04.692363 20164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:58:04.703478 20164 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:04.704377 20164 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:04.711747 20164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb: Generated new TSK 0
I20260928 04:58:04.712013 20164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:04.720096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44043
--builtin_ntp_servers=127.0.98.20:45335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:58:04.835880 20178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:04.836133 20178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:04.836161 20178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:04.837482 20178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:04.837540 20178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:04.837569 20178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:58:04.839329 20178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:04.839628 20178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:04.839978 20178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:04.846036 20186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:04.846025 20183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:04.846025 20184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:04.846575 20178 server_base.cc:1034] running on GCE node
I20260928 04:58:04.846854 20178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:04.847134 20178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:04.848287 20178 hybrid_clock.cc:648] HybridClock initialized: now 1790571484848262 us; error 42 us; skew 500 ppm
I20260928 04:58:04.849944 20178 webserver.cc:460] Webserver started at http://127.0.98.1:38371/ using document root <none> and password file <none>
I20260928 04:58:04.850324 20178 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:04.850382 20178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:04.850584 20178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:04.851963 20178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-0/data/instance:
uuid: "d0708ad3260641bb8f6480e72f7c2b49"
format_stamp: "Formatted at 2026-09-28 04:58:04 on dist-test-slave-m39l"
server_key: "63e4dc8e8c918dbb1fb0481369a271dc"
server_key_iv: "567b40d7f790f2e610b81750bb76b870"
server_key_version: "encryptionkey@0"
I20260928 04:58:04.852408 20178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d0708ad3260641bb8f6480e72f7c2b49"
format_stamp: "Formatted at 2026-09-28 04:58:04 on dist-test-slave-m39l"
server_key: "63e4dc8e8c918dbb1fb0481369a271dc"
server_key_iv: "567b40d7f790f2e610b81750bb76b870"
server_key_version: "encryptionkey@0"
I20260928 04:58:04.854146 20178 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 04:58:04.855255 20192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:04.855528 20178 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:04.855657 20178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-0/wal
uuid: "d0708ad3260641bb8f6480e72f7c2b49"
format_stamp: "Formatted at 2026-09-28 04:58:04 on dist-test-slave-m39l"
server_key: "63e4dc8e8c918dbb1fb0481369a271dc"
server_key_iv: "567b40d7f790f2e610b81750bb76b870"
server_key_version: "encryptionkey@0"
I20260928 04:58:04.855764 20178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:04.877295 20178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:04.877684 20178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:04.877844 20178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:04.878176 20178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:58:04.878587 20178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:58:04.878638 20178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:04.878680 20178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:58:04.878708 20178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:04.884796 20178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44785
I20260928 04:58:04.884855 20305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44785 every 8 connection(s)
I20260928 04:58:04.885238 20178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:58:04.886740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 20178
I20260928 04:58:04.886852 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:58:04.887042 392 external_mini_cluster.cc:1428] Setting key 49cef6a4a6bba791359a623943885bf6
I20260928 04:58:04.888213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44043
--builtin_ntp_servers=127.0.98.20:45335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 04:58:04.892143 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.886120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47717 (local address 127.0.98.62:44043)
0928 04:58:04.886208 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:04.886210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:04.886449 (+ 239us) server_negotiation.cc:407] Connection header received
0928 04:58:04.887257 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:04.887258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:04.887270 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:04.887341 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:04.888295 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.889207 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.890440 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.890721 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.890812 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.891826 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:04.891828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:04.891828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:04.891833 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:04.891848 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:04.891852 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:04.891923 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:04.892025 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 04:58:04.892054 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 04:58:04.892593 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:04.886094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44043 (local address 127.0.98.1:47717)
0928 04:58:04.886292 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:04.886338 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:04.887116 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:04.887433 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:04.887437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:04.887448 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:04.888188 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:04.888195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.889313 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.889314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:04.890329 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:04.890332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:04.890826 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:04.890827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:04.890970 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:04.891606 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:58:04.891606 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:04.891638 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:58:04.891665 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:04.891915 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:04.891923 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:04.891928 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:58:04.892020 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 04:58:04.892186 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260928 04:58:04.893167 20306 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44043
I20260928 04:58:04.893296 20306 heartbeater.cc:464] Registering TS with master...
I20260928 04:58:04.893576 20306 heartbeater.cc:511] Master 127.0.98.62:44043 requested a full tablet report, sending...
I20260928 04:58:04.894150 20118 ts_manager.cc:194] Registered new tserver with Master: d0708ad3260641bb8f6480e72f7c2b49 (127.0.98.1:44785)
I20260928 04:58:04.895017 20118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47717
W20260928 04:58:04.977612 20309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:04.977895 20309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:04.977938 20309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:04.979364 20309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:04.979436 20309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:04.979490 20309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:58:04.980989 20309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:04.981287 20309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:04.981612 20309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:04.986898 20314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:04.987027 20309 server_base.cc:1034] running on GCE node
W20260928 04:58:04.986903 20315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:04.986898 20317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:04.987424 20309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:04.987694 20309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:04.988871 20309 hybrid_clock.cc:648] HybridClock initialized: now 1790571484988814 us; error 73 us; skew 500 ppm
I20260928 04:58:04.990322 20309 webserver.cc:460] Webserver started at http://127.0.98.2:43117/ using document root <none> and password file <none>
I20260928 04:58:04.990658 20309 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:04.990731 20309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:04.990892 20309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:04.992053 20309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-1/data/instance:
uuid: "ae5923ab019740ce866dc519b9a3b911"
format_stamp: "Formatted at 2026-09-28 04:58:04 on dist-test-slave-m39l"
server_key: "567e5ace93512fea7d2cbdec06849a6a"
server_key_iv: "74ebab169a98228a56344065506e9fbc"
server_key_version: "encryptionkey@0"
I20260928 04:58:04.992432 20309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ae5923ab019740ce866dc519b9a3b911"
format_stamp: "Formatted at 2026-09-28 04:58:04 on dist-test-slave-m39l"
server_key: "567e5ace93512fea7d2cbdec06849a6a"
server_key_iv: "74ebab169a98228a56344065506e9fbc"
server_key_version: "encryptionkey@0"
I20260928 04:58:04.993850 20309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:58:04.994758 20323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:04.995031 20309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:04.995198 20309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-1/wal
uuid: "ae5923ab019740ce866dc519b9a3b911"
format_stamp: "Formatted at 2026-09-28 04:58:04 on dist-test-slave-m39l"
server_key: "567e5ace93512fea7d2cbdec06849a6a"
server_key_iv: "74ebab169a98228a56344065506e9fbc"
server_key_version: "encryptionkey@0"
I20260928 04:58:04.995286 20309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:05.034159 20309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:05.034579 20309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:05.034746 20309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:05.035005 20309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:58:05.035372 20309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:58:05.035429 20309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:05.035472 20309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:58:05.035498 20309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:05.041919 20309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38871
I20260928 04:58:05.042006 20436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38871 every 8 connection(s)
I20260928 04:58:05.042481 20309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:58:05.044495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 20309
I20260928 04:58:05.044610 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:58:05.044857 392 external_mini_cluster.cc:1428] Setting key 7c5470e4b97b05c0570697c62caeb040
I20260928 04:58:05.046023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44043
--builtin_ntp_servers=127.0.98.20:45335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 04:58:05.049484 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:05.043641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42587 (local address 127.0.98.62:44043)
0928 04:58:05.043721 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:05.043725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:05.043931 (+ 206us) server_negotiation.cc:407] Connection header received
0928 04:58:05.044700 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:05.044702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:05.044718 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:05.044753 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:05.045870 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.046592 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.047961 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.048164 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.048248 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:05.049168 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:05.049169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:05.049170 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:05.049176 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:05.049197 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:05.049201 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:05.049256 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:05.049360 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 04:58:05.049395 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 04:58:05.049839 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:05.043587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44043 (local address 127.0.98.2:42587)
0928 04:58:05.043811 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:05.043848 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:05.044537 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:05.044974 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:05.044979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:05.044991 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:05.045758 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:05.045767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.046702 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.046703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:05.047860 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:05.047863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.048255 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.048257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:05.048405 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:05.048993 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:58:05.048994 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:05.049028 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:58:05.049055 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:05.049269 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:05.049275 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:05.049279 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:58:05.049365 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 04:58:05.049485 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":112,"threads_started":1}
I20260928 04:58:05.050326 20437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44043
I20260928 04:58:05.050422 20437 heartbeater.cc:464] Registering TS with master...
I20260928 04:58:05.050689 20437 heartbeater.cc:511] Master 127.0.98.62:44043 requested a full tablet report, sending...
I20260928 04:58:05.051164 20118 ts_manager.cc:194] Registered new tserver with Master: ae5923ab019740ce866dc519b9a3b911 (127.0.98.2:38871)
I20260928 04:58:05.051556 20118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:42587
W20260928 04:58:05.140151 20440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:05.140440 20440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:05.140486 20440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:05.142112 20440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:05.142204 20440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:05.142261 20440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:58:05.144070 20440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:05.144411 20440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:05.144779 20440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:05.150767 20446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:05.150789 20448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:05.150767 20445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:05.151279 20440 server_base.cc:1034] running on GCE node
I20260928 04:58:05.151564 20440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:05.151880 20440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:05.153028 20440 hybrid_clock.cc:648] HybridClock initialized: now 1790571485153046 us; error 64 us; skew 500 ppm
I20260928 04:58:05.154386 20440 webserver.cc:460] Webserver started at http://127.0.98.3:33105/ using document root <none> and password file <none>
I20260928 04:58:05.154779 20440 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:05.154842 20440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:05.154987 20440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:05.156095 20440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-2/data/instance:
uuid: "e1ff7b96e41943aba49d8c4ea0ab8f84"
format_stamp: "Formatted at 2026-09-28 04:58:05 on dist-test-slave-m39l"
server_key: "7601d48d933dc4f983522f15c1fc06fb"
server_key_iv: "7aa8bae5b330b665213fd94fe24d7f87"
server_key_version: "encryptionkey@0"
I20260928 04:58:05.156454 20440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e1ff7b96e41943aba49d8c4ea0ab8f84"
format_stamp: "Formatted at 2026-09-28 04:58:05 on dist-test-slave-m39l"
server_key: "7601d48d933dc4f983522f15c1fc06fb"
server_key_iv: "7aa8bae5b330b665213fd94fe24d7f87"
server_key_version: "encryptionkey@0"
I20260928 04:58:05.157815 20440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:58:05.158742 20454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:05.158951 20440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:05.159031 20440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-2/wal
uuid: "e1ff7b96e41943aba49d8c4ea0ab8f84"
format_stamp: "Formatted at 2026-09-28 04:58:05 on dist-test-slave-m39l"
server_key: "7601d48d933dc4f983522f15c1fc06fb"
server_key_iv: "7aa8bae5b330b665213fd94fe24d7f87"
server_key_version: "encryptionkey@0"
I20260928 04:58:05.159121 20440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:05.182753 20440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:05.183075 20440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:05.183195 20440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:05.183502 20440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:58:05.183835 20440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:58:05.183869 20440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:05.183925 20440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:58:05.183959 20440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:05.190871 20440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36017
I20260928 04:58:05.190914 20567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36017 every 8 connection(s)
I20260928 04:58:05.191322 20440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:58:05.192021 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 20440
I20260928 04:58:05.192147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:58:05.192384 392 external_mini_cluster.cc:1428] Setting key 5c2bfea7b917eed3a978053febd62cd1
I20260928 04:58:05.197939 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:05.192593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49897 (local address 127.0.98.62:44043)
0928 04:58:05.192666 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:05.192668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:05.192678 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:58:05.193402 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:05.193403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:05.193418 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:05.193542 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:05.194518 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.195072 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.196233 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.196475 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.196562 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:05.197600 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:05.197602 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:05.197602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:05.197609 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:05.197627 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:05.197631 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:05.197706 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:05.197807 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 04:58:05.197843 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 04:58:05.198227 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:05.192182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44043 (local address 127.0.98.3:49897)
0928 04:58:05.192417 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:05.192476 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:05.193272 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:05.193644 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:05.193648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:05.193658 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:05.194351 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:05.194358 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.195197 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.195198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:05.196121 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:05.196124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.196587 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.196588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:05.196737 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:05.197415 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:58:05.197416 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:05.197451 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:58:05.197482 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:05.197707 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:05.197714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:05.197718 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:58:05.197811 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 04:58:05.197938 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":94,"threads_started":1}
I20260928 04:58:05.198758 20568 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44043
I20260928 04:58:05.198863 20568 heartbeater.cc:464] Registering TS with master...
I20260928 04:58:05.199086 20568 heartbeater.cc:511] Master 127.0.98.62:44043 requested a full tablet report, sending...
I20260928 04:58:05.199498 20118 ts_manager.cc:194] Registered new tserver with Master: e1ff7b96e41943aba49d8c4ea0ab8f84 (127.0.98.3:36017)
I20260928 04:58:05.199925 20118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:49897
I20260928 04:58:05.204711 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:58:05.208912 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:05.206116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39250 (local address 127.0.98.62:44043)
0928 04:58:05.206171 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:05.206173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:05.206224 (+ 51us) server_negotiation.cc:407] Connection header received
0928 04:58:05.206279 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:05.206280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:05.206288 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:05.206342 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:05.206548 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.207028 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.207853 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.208056 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.208115 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:05.208716 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:05.208717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:05.208717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:05.208722 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:05.208736 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:05.208739 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:05.208782 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:05.208827 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 04:58:05.208849 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:58:05.210708 20118 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:58:05.212472 20118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:58:05.219507 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:05.215056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44785 (local address 127.0.0.1:36390)
0928 04:58:05.215430 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:05.215442 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:05.215504 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:05.215813 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:05.215816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:05.215823 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:05.215939 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:05.215942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.216964 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.216965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:05.218294 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:05.218297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.219148 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.219149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:05.219253 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:05.219269 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 04:58:05.219354 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 04:58:05.219426 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":4,"thread_start_us":63,"threads_started":1}
I20260928 04:58:05.219525 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:05.215235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.0.98.1:44785)
0928 04:58:05.215563 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:05.215566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:05.215577 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:58:05.215625 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:05.215627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:05.215644 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:05.215726 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:05.216033 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.216835 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.218415 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.219042 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.219149 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:05.219261 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:05.219370 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 04:58:05.219414 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":93,"threads_started":1}
I20260928 04:58:05.219667 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:05.215054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38871 (local address 127.0.0.1:58430)
0928 04:58:05.215340 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:05.215388 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:05.215462 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:05.215812 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:05.215815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:05.215818 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:05.215947 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:05.215949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.217346 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.217347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:05.218444 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:05.218446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.219406 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.219407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:05.219477 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:05.219479 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:58:05.219525 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 04:58:05.219581 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":64,"thread_start_us":112,"threads_started":1}
I20260928 04:58:05.219764 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:05.215290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59574 (local address 127.0.98.3:36017)
0928 04:58:05.215526 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:05.215532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:05.215782 (+ 250us) server_negotiation.cc:407] Connection header received
0928 04:58:05.215833 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:05.215836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:05.215852 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:05.215948 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:05.216132 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.217243 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.218386 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.219190 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.219269 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:05.219364 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:05.219628 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 04:58:05.219668 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":76,"threads_started":1}
I20260928 04:58:05.219955 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:05.215234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58430 (local address 127.0.98.2:38871)
0928 04:58:05.215461 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:05.215466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:05.215480 (+ 14us) server_negotiation.cc:407] Connection header received
0928 04:58:05.215606 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:05.215609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:05.215625 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:05.215724 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:05.216048 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.217244 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.218561 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.219244 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.219303 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:05.219378 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:05.219840 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 04:58:05.219891 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":90,"threads_started":1}
I20260928 04:58:05.220175 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:05.215054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36017 (local address 127.0.0.1:59574)
0928 04:58:05.215613 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:05.215623 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:05.215699 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:05.216024 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:05.216024 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:05.216027 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:05.216062 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:05.216064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.217355 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.217356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:05.218291 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:05.218294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.219403 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.219404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:05.219467 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:05.219472 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:58:05.219525 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 04:58:05.219581 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":15,"spinlock_wait_cycles":896,"thread_start_us":106,"threads_started":1}
I20260928 04:58:05.220826 20502 tablet_service.cc:1467] Processing CreateTablet for tablet f97fe302bb23438ca115eb4d79a06e00 (DEFAULT_TABLE table=test-table [id=9f222b4a0508449189da9aa60ac860a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:58:05.220824 20371 tablet_service.cc:1467] Processing CreateTablet for tablet f97fe302bb23438ca115eb4d79a06e00 (DEFAULT_TABLE table=test-table [id=9f222b4a0508449189da9aa60ac860a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:58:05.221154 20502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f97fe302bb23438ca115eb4d79a06e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:05.221251 20371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f97fe302bb23438ca115eb4d79a06e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:05.224202 20585 tablet_bootstrap.cc:492] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84: Bootstrap starting.
I20260928 04:58:05.224668 20240 tablet_service.cc:1467] Processing CreateTablet for tablet f97fe302bb23438ca115eb4d79a06e00 (DEFAULT_TABLE table=test-table [id=9f222b4a0508449189da9aa60ac860a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:58:05.224982 20585 tablet_bootstrap.cc:654] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:05.225020 20240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f97fe302bb23438ca115eb4d79a06e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:05.225317 20585 log.cc:824] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:05.226087 20585 tablet_bootstrap.cc:492] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84: No bootstrap required, opened a new log
I20260928 04:58:05.226153 20585 ts_tablet_manager.cc:1397] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:58:05.226694 20586 tablet_bootstrap.cc:492] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911: Bootstrap starting.
I20260928 04:58:05.227396 20586 tablet_bootstrap.cc:654] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:05.227758 20586 log.cc:824] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:05.228296 20589 tablet_bootstrap.cc:492] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49: Bootstrap starting.
I20260928 04:58:05.228644 20586 tablet_bootstrap.cc:492] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911: No bootstrap required, opened a new log
I20260928 04:58:05.228729 20586 ts_tablet_manager.cc:1397] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:58:05.228938 20589 tablet_bootstrap.cc:654] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:05.229221 20589 log.cc:824] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:05.229348 20585 raft_consensus.cc:348] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5923ab019740ce866dc519b9a3b911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38871 } } peers { permanent_uuid: "e1ff7b96e41943aba49d8c4ea0ab8f84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } peers { permanent_uuid: "d0708ad3260641bb8f6480e72f7c2b49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44785 } }
I20260928 04:58:05.229606 20585 raft_consensus.cc:374] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:05.229658 20585 raft_consensus.cc:729] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1ff7b96e41943aba49d8c4ea0ab8f84, State: Initialized, Role: FOLLOWER
I20260928 04:58:05.229797 20585 consensus_queue.cc:260] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5923ab019740ce866dc519b9a3b911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38871 } } peers { permanent_uuid: "e1ff7b96e41943aba49d8c4ea0ab8f84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } peers { permanent_uuid: "d0708ad3260641bb8f6480e72f7c2b49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44785 } }
I20260928 04:58:05.230087 20585 ts_tablet_manager.cc:1428] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 04:58:05.230162 20589 tablet_bootstrap.cc:492] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49: No bootstrap required, opened a new log
I20260928 04:58:05.230238 20589 ts_tablet_manager.cc:1397] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:58:05.230237 20568 heartbeater.cc:503] Master 127.0.98.62:44043 was elected leader, sending a full tablet report...
I20260928 04:58:05.231709 20586 raft_consensus.cc:348] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5923ab019740ce866dc519b9a3b911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38871 } } peers { permanent_uuid: "e1ff7b96e41943aba49d8c4ea0ab8f84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } peers { permanent_uuid: "d0708ad3260641bb8f6480e72f7c2b49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44785 } }
I20260928 04:58:05.231909 20586 raft_consensus.cc:374] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:05.231938 20586 raft_consensus.cc:729] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5923ab019740ce866dc519b9a3b911, State: Initialized, Role: FOLLOWER
I20260928 04:58:05.232065 20586 consensus_queue.cc:260] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5923ab019740ce866dc519b9a3b911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38871 } } peers { permanent_uuid: "e1ff7b96e41943aba49d8c4ea0ab8f84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } peers { permanent_uuid: "d0708ad3260641bb8f6480e72f7c2b49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44785 } }
I20260928 04:58:05.232357 20586 ts_tablet_manager.cc:1428] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 04:58:05.232445 20437 heartbeater.cc:503] Master 127.0.98.62:44043 was elected leader, sending a full tablet report...
I20260928 04:58:05.232887 20589 raft_consensus.cc:348] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5923ab019740ce866dc519b9a3b911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38871 } } peers { permanent_uuid: "e1ff7b96e41943aba49d8c4ea0ab8f84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } peers { permanent_uuid: "d0708ad3260641bb8f6480e72f7c2b49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44785 } }
I20260928 04:58:05.233067 20589 raft_consensus.cc:374] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:05.233114 20589 raft_consensus.cc:729] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0708ad3260641bb8f6480e72f7c2b49, State: Initialized, Role: FOLLOWER
I20260928 04:58:05.233251 20589 consensus_queue.cc:260] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5923ab019740ce866dc519b9a3b911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38871 } } peers { permanent_uuid: "e1ff7b96e41943aba49d8c4ea0ab8f84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } peers { permanent_uuid: "d0708ad3260641bb8f6480e72f7c2b49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44785 } }
I20260928 04:58:05.233554 20589 ts_tablet_manager.cc:1428] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 04:58:05.233664 20306 heartbeater.cc:503] Master 127.0.98.62:44043 was elected leader, sending a full tablet report...
I20260928 04:58:05.264822 20593 raft_consensus.cc:482] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:58:05.264972 20593 raft_consensus.cc:504] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5923ab019740ce866dc519b9a3b911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38871 } } peers { permanent_uuid: "e1ff7b96e41943aba49d8c4ea0ab8f84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } peers { permanent_uuid: "d0708ad3260641bb8f6480e72f7c2b49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44785 } }
I20260928 04:58:05.265326 20593 leader_election.cc:288] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae5923ab019740ce866dc519b9a3b911 (127.0.98.2:38871), e1ff7b96e41943aba49d8c4ea0ab8f84 (127.0.98.3:36017)
I20260928 04:58:05.268415 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:05.265658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46513 (local address 127.0.98.2:38871)
0928 04:58:05.265712 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:05.265714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:05.265723 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:58:05.265756 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:05.265757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:05.265769 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:05.265854 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:05.266038 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.266636 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.267701 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.268057 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.268131 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:05.268143 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:05.268291 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 04:58:05.268336 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 04:58:05.268445 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:05.265628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57685 (local address 127.0.98.3:36017)
0928 04:58:05.265700 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:05.265702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:05.265867 (+ 165us) server_negotiation.cc:407] Connection header received
0928 04:58:05.265907 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:05.265908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:05.265918 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:05.265981 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:58:05.266216 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.266773 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.267697 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.268060 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.268147 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:05.268156 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:05.268329 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 04:58:05.268373 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 04:58:05.268445 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:05.265543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38871 (local address 127.0.98.1:46513)
0928 04:58:05.265608 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:05.265621 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:05.265671 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:05.265876 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:05.265878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:05.265880 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:05.265960 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:05.265962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.266714 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.266716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:05.267569 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:05.267573 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.268140 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.268141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:05.268218 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:05.268221 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:58:05.268264 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 04:58:05.268320 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":1}
I20260928 04:58:05.268455 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:05.265543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36017 (local address 127.0.98.1:57685)
0928 04:58:05.265784 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:05.265796 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:05.265856 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:05.266008 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:05.266009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:05.266011 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:58:05.266086 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:05.266087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.266855 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.266855 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:05.267596 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:05.267598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.268160 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.268161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:05.268235 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:05.268238 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:58:05.268310 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 04:58:05.268345 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":22,"thread_start_us":133,"threads_started":1}
I20260928 04:58:05.268839 20391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f97fe302bb23438ca115eb4d79a06e00" candidate_uuid: "d0708ad3260641bb8f6480e72f7c2b49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5923ab019740ce866dc519b9a3b911" is_pre_election: true
I20260928 04:58:05.269011 20391 raft_consensus.cc:2454] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0708ad3260641bb8f6480e72f7c2b49 in term 0.
I20260928 04:58:05.269291 20193 leader_election.cc:302] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae5923ab019740ce866dc519b9a3b911, d0708ad3260641bb8f6480e72f7c2b49; no voters:
I20260928 04:58:05.269316 20522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f97fe302bb23438ca115eb4d79a06e00" candidate_uuid: "d0708ad3260641bb8f6480e72f7c2b49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1ff7b96e41943aba49d8c4ea0ab8f84" is_pre_election: true
I20260928 04:58:05.269446 20522 raft_consensus.cc:2454] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0708ad3260641bb8f6480e72f7c2b49 in term 0.
I20260928 04:58:05.269474 20593 raft_consensus.cc:2781] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:58:05.269548 20593 raft_consensus.cc:482] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:58:05.269610 20593 raft_consensus.cc:3037] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:05.270360 20593 raft_consensus.cc:504] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5923ab019740ce866dc519b9a3b911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38871 } } peers { permanent_uuid: "e1ff7b96e41943aba49d8c4ea0ab8f84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } peers { permanent_uuid: "d0708ad3260641bb8f6480e72f7c2b49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44785 } }
I20260928 04:58:05.270533 20593 leader_election.cc:288] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [CANDIDATE]: Term 1 election: Requested vote from peers ae5923ab019740ce866dc519b9a3b911 (127.0.98.2:38871), e1ff7b96e41943aba49d8c4ea0ab8f84 (127.0.98.3:36017)
I20260928 04:58:05.270711 20522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f97fe302bb23438ca115eb4d79a06e00" candidate_uuid: "d0708ad3260641bb8f6480e72f7c2b49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1ff7b96e41943aba49d8c4ea0ab8f84"
I20260928 04:58:05.270735 20391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f97fe302bb23438ca115eb4d79a06e00" candidate_uuid: "d0708ad3260641bb8f6480e72f7c2b49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5923ab019740ce866dc519b9a3b911"
I20260928 04:58:05.270817 20522 raft_consensus.cc:3037] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:05.270812 20391 raft_consensus.cc:3037] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:05.271632 20522 raft_consensus.cc:2454] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0708ad3260641bb8f6480e72f7c2b49 in term 1.
I20260928 04:58:05.271637 20391 raft_consensus.cc:2454] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0708ad3260641bb8f6480e72f7c2b49 in term 1.
I20260928 04:58:05.271862 20193 leader_election.cc:302] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae5923ab019740ce866dc519b9a3b911, d0708ad3260641bb8f6480e72f7c2b49; no voters:
I20260928 04:58:05.271978 20593 raft_consensus.cc:2781] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:05.272161 20593 raft_consensus.cc:686] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [term 1 LEADER]: Becoming Leader. State: Replica: d0708ad3260641bb8f6480e72f7c2b49, State: Running, Role: LEADER
I20260928 04:58:05.272277 20593 consensus_queue.cc:237] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5923ab019740ce866dc519b9a3b911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38871 } } peers { permanent_uuid: "e1ff7b96e41943aba49d8c4ea0ab8f84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } peers { permanent_uuid: "d0708ad3260641bb8f6480e72f7c2b49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44785 } }
I20260928 04:58:05.273279 20118 catalog_manager.cc:5495] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0708ad3260641bb8f6480e72f7c2b49 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d0708ad3260641bb8f6480e72f7c2b49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5923ab019740ce866dc519b9a3b911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1ff7b96e41943aba49d8c4ea0ab8f84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0708ad3260641bb8f6480e72f7c2b49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44785 } health_report { overall_health: HEALTHY } } }
W20260928 04:58:05.293179 20438 tablet.cc:2369] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:58:05.294829 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:05.291987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.0.98.1:44785)
0928 04:58:05.292053 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:05.292055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:05.292134 (+ 79us) server_negotiation.cc:407] Connection header received
0928 04:58:05.292235 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:05.292236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:05.292257 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:05.292327 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:58:05.292450 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.292934 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.293904 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:05.294170 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:05.294255 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:05.294424 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:58:05.294608 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:58:05.294676 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:05.294726 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 04:58:05.294746 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:58:05.296945 20522 raft_consensus.cc:1261] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84 [term 1 FOLLOWER]: Refusing update from remote peer d0708ad3260641bb8f6480e72f7c2b49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:58:05.296945 20391 raft_consensus.cc:1261] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911 [term 1 FOLLOWER]: Refusing update from remote peer d0708ad3260641bb8f6480e72f7c2b49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:58:05.297400 20595 consensus_queue.cc:1035] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae5923ab019740ce866dc519b9a3b911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:58:05.297573 20593 consensus_queue.cc:1035] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1ff7b96e41943aba49d8c4ea0ab8f84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:58:05.300622 20598 mvcc.cc:205] Tried to move back new op lower bound from 7334180803772772352 to 7334180803675979776. Current Snapshot: MvccSnapshot[applied={T|T < 7334180803772772352}]
I20260928 04:58:05.301087 20601 mvcc.cc:205] Tried to move back new op lower bound from 7334180803772772352 to 7334180803675979776. Current Snapshot: MvccSnapshot[applied={T|T < 7334180803772772352}]
I20260928 04:58:05.301213 20599 mvcc.cc:205] Tried to move back new op lower bound from 7334180803772772352 to 7334180803675979776. Current Snapshot: MvccSnapshot[applied={T|T < 7334180803772772352}]
I20260928 04:58:05.303475 20117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:58:05.303599 20117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:58:05.304454 20117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d02e0b3345f48d099cb7460c22e35eb: Sending DeleteTablet for 3 replicas of tablet f97fe302bb23438ca115eb4d79a06e00
I20260928 04:58:05.304826 20371 tablet_service.cc:1514] Processing DeleteTablet for tablet f97fe302bb23438ca115eb4d79a06e00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:58:05 UTC) from {username='slave'} at 127.0.0.1:58430
I20260928 04:58:05.304847 20240 tablet_service.cc:1514] Processing DeleteTablet for tablet f97fe302bb23438ca115eb4d79a06e00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:58:05 UTC) from {username='slave'} at 127.0.0.1:36390
I20260928 04:58:05.304935 20502 tablet_service.cc:1514] Processing DeleteTablet for tablet f97fe302bb23438ca115eb4d79a06e00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:58:05 UTC) from {username='slave'} at 127.0.0.1:59574
I20260928 04:58:05.305330 20610 tablet_replica.cc:330] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84: stopping tablet replica
I20260928 04:58:05.305382 20609 tablet_replica.cc:330] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49: stopping tablet replica
I20260928 04:58:05.305470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 20178
I20260928 04:58:05.305526 20609 raft_consensus.cc:2229] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:58:05.305614 20610 raft_consensus.cc:2229] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:58:05.305660 20609 raft_consensus.cc:2258] T f97fe302bb23438ca115eb4d79a06e00 P d0708ad3260641bb8f6480e72f7c2b49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:58:05.305823 20610 raft_consensus.cc:2258] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:58:05.306576 20608 tablet_replica.cc:330] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911: stopping tablet replica
I20260928 04:58:05.306649 20610 ts_tablet_manager.cc:1905] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:58:05.306731 20608 raft_consensus.cc:2229] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:58:05.306864 20608 raft_consensus.cc:2258] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:58:05.307312 20608 ts_tablet_manager.cc:1905] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:58:05.308861 20610 ts_tablet_manager.cc:1918] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:58:05.308948 20610 log.cc:1196] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-2/wal/wals/f97fe302bb23438ca115eb4d79a06e00
I20260928 04:58:05.309242 20610 ts_tablet_manager.cc:1939] T f97fe302bb23438ca115eb4d79a06e00 P e1ff7b96e41943aba49d8c4ea0ab8f84: Deleting consensus metadata
I20260928 04:58:05.309219 20608 ts_tablet_manager.cc:1918] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:58:05.309406 20608 log.cc:1196] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911: Deleting WAL directory at /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790571208412331-392-0/minicluster-data/ts-1/wal/wals/f97fe302bb23438ca115eb4d79a06e00
I20260928 04:58:05.309762 20608 ts_tablet_manager.cc:1939] T f97fe302bb23438ca115eb4d79a06e00 P ae5923ab019740ce866dc519b9a3b911: Deleting consensus metadata
I20260928 04:58:05.309784 20104 catalog_manager.cc:4841] TS e1ff7b96e41943aba49d8c4ea0ab8f84 (127.0.98.3:36017): tablet f97fe302bb23438ca115eb4d79a06e00 (table test-table [id=9f222b4a0508449189da9aa60ac860a4]) successfully deleted
I20260928 04:58:05.310287 20102 catalog_manager.cc:4841] TS ae5923ab019740ce866dc519b9a3b911 (127.0.98.2:38871): tablet f97fe302bb23438ca115eb4d79a06e00 (table test-table [id=9f222b4a0508449189da9aa60ac860a4]) successfully deleted
W20260928 04:58:05.312983 20103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44785 (error 108)
I20260928 04:58:05.313273 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 20309
I20260928 04:58:05.313989 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:05.313449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44785 (local address 127.0.0.1:36402)
0928 04:58:05.313883 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:05.313942 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408}
W20260928 04:58:05.314270 20103 catalog_manager.cc:4584] TS d0708ad3260641bb8f6480e72f7c2b49 (127.0.98.1:44785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f97fe302bb23438ca115eb4d79a06e00: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44785: connect: Connection refused (error 111)
I20260928 04:58:05.318522 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 20440
I20260928 04:58:05.323992 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 20087
2026-09-28T04:58:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (805 ms)
[----------] 6 tests from EncryptionPolicyTest (9054 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:05 dist-test-slave-m39l krb5kdc[20614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:05 dist-test-slave-m39l krb5kdc[20614](info): set up 2 sockets
Sep 28 04:58:05 dist-test-slave-m39l krb5kdc[20614](info): commencing operation
krb5kdc: starting...
W20260928 04:58:06.386739 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:06 dist-test-slave-m39l krb5kdc[20614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:58:06.522936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41677
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:06.615278 20627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:06.615541 20627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:06.615569 20627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:06.616950 20627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:06.617019 20627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:06.617035 20627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:06.617048 20627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:06.617061 20627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:06.619218 20627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41677
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:06.619606 20627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:06.619913 20627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:06.625819 20632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:06.625834 20635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:06.625834 20633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:06.626266 20627 server_base.cc:1034] running on GCE node
I20260928 04:58:06.626616 20627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:06.626976 20627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:06.628168 20627 hybrid_clock.cc:648] HybridClock initialized: now 1790571486628144 us; error 36 us; skew 500 ppm
Sep 28 04:58:06 dist-test-slave-m39l krb5kdc[20614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:06.635759 20627 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:58:06.636399 20627 webserver.cc:460] Webserver started at http://127.0.0.1:44535/ using document root <none> and password file <none>
I20260928 04:58:06.636673 20627 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:06.636727 20627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:06.636888 20627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:06.638095 20627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "214339f16c5e46afa8264c616c323ff0"
format_stamp: "Formatted at 2026-09-28 04:58:06 on dist-test-slave-m39l"
server_key: "ce4262cdd723948b9b3d0a5d3025c4d9"
server_key_iv: "3a25ba7dddedcf9fbd1bc4e73121fbbf"
server_key_version: "encryptionkey@0"
I20260928 04:58:06.638592 20627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "214339f16c5e46afa8264c616c323ff0"
format_stamp: "Formatted at 2026-09-28 04:58:06 on dist-test-slave-m39l"
server_key: "ce4262cdd723948b9b3d0a5d3025c4d9"
server_key_iv: "3a25ba7dddedcf9fbd1bc4e73121fbbf"
server_key_version: "encryptionkey@0"
I20260928 04:58:06.640187 20627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:58:06.641184 20642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:06.641536 20627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 04:58:06.641647 20627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "214339f16c5e46afa8264c616c323ff0"
format_stamp: "Formatted at 2026-09-28 04:58:06 on dist-test-slave-m39l"
server_key: "ce4262cdd723948b9b3d0a5d3025c4d9"
server_key_iv: "3a25ba7dddedcf9fbd1bc4e73121fbbf"
server_key_version: "encryptionkey@0"
I20260928 04:58:06.641734 20627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:06.662544 20627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:06.664939 20627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:06.665165 20627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:06.669806 20627 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41677
I20260928 04:58:06.669917 20694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41677 every 8 connection(s)
I20260928 04:58:06.670290 20627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:06.671363 20695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:06.674273 20695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0: Bootstrap starting.
I20260928 04:58:06.674948 20695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:06.675400 20695 log.cc:824] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:06.676419 20695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0: No bootstrap required, opened a new log
I20260928 04:58:06.678296 20695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214339f16c5e46afa8264c616c323ff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41677 } }
I20260928 04:58:06.678478 20695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:06.678539 20695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214339f16c5e46afa8264c616c323ff0, State: Initialized, Role: FOLLOWER
I20260928 04:58:06.678718 20695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214339f16c5e46afa8264c616c323ff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41677 } }
I20260928 04:58:06.678818 20695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:06.678858 20695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:06.678941 20695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:06.679141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 20627
I20260928 04:58:06.679260 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:06.679473 392 external_mini_cluster.cc:1428] Setting key e46848e7fd09bea1b11720771a0feef3
I20260928 04:58:06.679896 20695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214339f16c5e46afa8264c616c323ff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41677 } }
I20260928 04:58:06.680100 20695 leader_election.cc:302] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 214339f16c5e46afa8264c616c323ff0; no voters:
I20260928 04:58:06.680310 20695 leader_election.cc:288] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:06.680344 20700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:06.680585 20700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0 [term 1 LEADER]: Becoming Leader. State: Replica: 214339f16c5e46afa8264c616c323ff0, State: Running, Role: LEADER
I20260928 04:58:06.680609 20695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:06.680756 20700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214339f16c5e46afa8264c616c323ff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41677 } }
I20260928 04:58:06.681589 20701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 214339f16c5e46afa8264c616c323ff0. Latest consensus state: current_term: 1 leader_uuid: "214339f16c5e46afa8264c616c323ff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214339f16c5e46afa8264c616c323ff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41677 } } }
I20260928 04:58:06.681703 20701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:06.682257 20709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:06.682358 20702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "214339f16c5e46afa8264c616c323ff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214339f16c5e46afa8264c616c323ff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41677 } } }
I20260928 04:58:06.682708 20702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:06.682963 20709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:06.684980 20709 catalog_manager.cc:1347] Generated new cluster ID: 390516c96d0f4787915e6968db2297ab
I20260928 04:58:06.685052 20709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:58:06 dist-test-slave-m39l krb5kdc[20614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 04:58:06.693190 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:06.680059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44221 (local address 127.0.0.1:41677)
0928 04:58:06.680409 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:06.680420 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:06.680462 (+ 42us) server_negotiation.cc:407] Connection header received
0928 04:58:06.680598 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:06.680606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:06.680670 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:06.680900 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:06.681856 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:06.683611 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:06.684618 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:06.685351 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:06.685478 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:06.688732 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:06.688737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:06.688738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:06.688756 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:06.691197 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:06.691693 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:06.691697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:06.691698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:06.691854 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:06.692115 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:06.692116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:06.692116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:06.692499 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:06.692697 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:06.692796 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 04:58:06.692917 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":137,"threads_started":1}
I20260928 04:58:06.694389 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:06.701009 20709 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:06.701826 20709 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:06.706193 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:06.696141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54969 (local address 127.0.0.1:41677)
0928 04:58:06.696184 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:06.696186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:06.698768 (+ 2582us) server_negotiation.cc:407] Connection header received
0928 04:58:06.698833 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:06.698835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:06.698850 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:06.698930 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:06.699626 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:06.700511 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:06.701501 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:06.701747 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:06.701834 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:06.703346 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:06.703348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:06.703348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:06.703356 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:06.704894 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:06.705320 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:06.705321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:06.705322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:06.705394 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:06.705628 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:06.705629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:06.705629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:06.705757 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:06.705896 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:06.706001 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 04:58:06.706067 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:58:06.723265 20709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 214339f16c5e46afa8264c616c323ff0: Generated new TSK 0
I20260928 04:58:06.723546 20709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:06.730038 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 20627
W20260928 04:58:06.734288 20612 connection.cc:535] client connection to 127.0.0.1:41677 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41677): Connection reset by peer (error 104)
2026-09-28T04:58:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:06 dist-test-slave-m39l krb5kdc[20726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:06 dist-test-slave-m39l krb5kdc[20726](info): set up 2 sockets
Sep 28 04:58:06 dist-test-slave-m39l krb5kdc[20726](info): commencing operation
krb5kdc: starting...
W20260928 04:58:07.827126 392 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:07 dist-test-slave-m39l krb5kdc[20726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:58:07.959327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:08.047008 20739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:08.047235 20739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:08.047259 20739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:08.048542 20739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:08.048597 20739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:08.048612 20739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:08.048625 20739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:08.048637 20739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:08.050213 20739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44239
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:08.050473 20739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:08.050782 20739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:08.055886 20747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:08.055873 20745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:08.055903 20739 server_base.cc:1034] running on GCE node
W20260928 04:58:08.055874 20744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:08.056355 20739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:08.056689 20739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:08.057837 20739 hybrid_clock.cc:648] HybridClock initialized: now 1790571488057820 us; error 39 us; skew 500 ppm
Sep 28 04:58:08 dist-test-slave-m39l krb5kdc[20726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:08.060592 20739 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:58:08.061106 20739 webserver.cc:460] Webserver started at http://127.0.0.1:37245/ using document root <none> and password file <none>
I20260928 04:58:08.061353 20739 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:08.061419 20739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:08.061558 20739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:08.062772 20739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "04257f1ea669430897691b98c69b194a"
format_stamp: "Formatted at 2026-09-28 04:58:08 on dist-test-slave-m39l"
server_key: "fdf0eda93526b3d31077320d8d69a67f"
server_key_iv: "e67ff3df69c53f238527798d1ae3b362"
server_key_version: "encryptionkey@0"
I20260928 04:58:08.063139 20739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "04257f1ea669430897691b98c69b194a"
format_stamp: "Formatted at 2026-09-28 04:58:08 on dist-test-slave-m39l"
server_key: "fdf0eda93526b3d31077320d8d69a67f"
server_key_iv: "e67ff3df69c53f238527798d1ae3b362"
server_key_version: "encryptionkey@0"
I20260928 04:58:08.064561 20739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:08.065371 20754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:08.065555 20739 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:08.065675 20739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "04257f1ea669430897691b98c69b194a"
format_stamp: "Formatted at 2026-09-28 04:58:08 on dist-test-slave-m39l"
server_key: "fdf0eda93526b3d31077320d8d69a67f"
server_key_iv: "e67ff3df69c53f238527798d1ae3b362"
server_key_version: "encryptionkey@0"
I20260928 04:58:08.065749 20739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:08.080164 20739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:08.082665 20739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:08.082849 20739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:08.087184 20739 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44239
I20260928 04:58:08.087239 20806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44239 every 8 connection(s)
I20260928 04:58:08.087788 20739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:08.088518 20807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:08.091049 20807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a: Bootstrap starting.
I20260928 04:58:08.091619 20807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:08.091864 20807 log.cc:824] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:08.092604 20807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a: No bootstrap required, opened a new log
I20260928 04:58:08.094219 20807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04257f1ea669430897691b98c69b194a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44239 } }
I20260928 04:58:08.094384 20807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:08.094408 20807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04257f1ea669430897691b98c69b194a, State: Initialized, Role: FOLLOWER
I20260928 04:58:08.094637 20807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04257f1ea669430897691b98c69b194a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44239 } }
I20260928 04:58:08.094712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 20739
I20260928 04:58:08.094748 20807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:08.094787 20807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:08.094826 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:08.094837 20807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:08.095041 392 external_mini_cluster.cc:1428] Setting key d7dac7831f0c99f93a5d1827a7438c55
I20260928 04:58:08.096227 20807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04257f1ea669430897691b98c69b194a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44239 } }
I20260928 04:58:08.096356 20807 leader_election.cc:302] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04257f1ea669430897691b98c69b194a; no voters:
I20260928 04:58:08.096555 20807 leader_election.cc:288] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:08.096625 20812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:08.096838 20812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a [term 1 LEADER]: Becoming Leader. State: Replica: 04257f1ea669430897691b98c69b194a, State: Running, Role: LEADER
I20260928 04:58:08.096854 20807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:08.096948 20812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04257f1ea669430897691b98c69b194a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44239 } }
I20260928 04:58:08.097342 20813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04257f1ea669430897691b98c69b194a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04257f1ea669430897691b98c69b194a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44239 } } }
I20260928 04:58:08.097512 20813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:08.097330 20812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04257f1ea669430897691b98c69b194a. Latest consensus state: current_term: 1 leader_uuid: "04257f1ea669430897691b98c69b194a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04257f1ea669430897691b98c69b194a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44239 } } }
I20260928 04:58:08.097733 20812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:08.097780 20821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:08.098380 20821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:08.101159 20821 catalog_manager.cc:1347] Generated new cluster ID: 662c839a02734ab385de9573bdd85dc9
I20260928 04:58:08.101346 20821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:58:08.101536 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:08.095612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51325 (local address 127.0.0.1:44239)
0928 04:58:08.095819 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:08.095827 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:08.095860 (+ 33us) server_negotiation.cc:407] Connection header received
0928 04:58:08.095986 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:08.095992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:08.096040 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:08.096159 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:08.096442 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:08.098035 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:08.099204 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:08.099761 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:08.099886 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:08.100676 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:08.100680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:08.100680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:08.100934 (+ 254us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:08.100977 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:08.100998 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:08.101093 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:08.101223 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 04:58:08.101256 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":110,"threads_started":1}
I20260928 04:58:08.102574 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:08.110594 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:08.104162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45693 (local address 127.0.0.1:44239)
0928 04:58:08.104201 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:08.104203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:08.106793 (+ 2590us) server_negotiation.cc:407] Connection header received
0928 04:58:08.106867 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:08.106869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:08.106883 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:08.106944 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:08.107126 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:08.107910 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:08.108984 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:08.109354 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:08.109434 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:08.110259 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:08.110261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:08.110262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:08.110268 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:08.110289 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:08.110294 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:08.110363 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:08.110436 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 04:58:08.110451 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:58:08.111425 20821 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:08.111933 20821 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:08.116279 20821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04257f1ea669430897691b98c69b194a: Generated new TSK 0
I20260928 04:58:08.116513 20821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:08.119004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 20739
W20260928 04:58:08.123315 20724 connection.cc:535] client connection to 127.0.0.1:44239 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44239): Connection reset by peer (error 104)
2026-09-28T04:58:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:08 dist-test-slave-m39l krb5kdc[20838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:08 dist-test-slave-m39l krb5kdc[20838](info): set up 2 sockets
Sep 28 04:58:08 dist-test-slave-m39l krb5kdc[20838](info): commencing operation
krb5kdc: starting...
W20260928 04:58:09.178711 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:09 dist-test-slave-m39l krb5kdc[20838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:58:09.306042 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37951
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:09.395004 20851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:09.395236 20851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:09.395260 20851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:09.396430 20851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:09.396476 20851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:09.396490 20851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:09.396503 20851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:09.396515 20851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:09.397975 20851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37951
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:09.398185 20851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:09.398370 20851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:09.403589 20859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:09.403568 20856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:09.403568 20857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:09.403599 20851 server_base.cc:1034] running on GCE node
I20260928 04:58:09.404091 20851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:09.404395 20851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:09.405586 20851 hybrid_clock.cc:648] HybridClock initialized: now 1790571489405572 us; error 30 us; skew 500 ppm
Sep 28 04:58:09 dist-test-slave-m39l krb5kdc[20838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:09.408298 20851 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:58:09.408782 20851 webserver.cc:460] Webserver started at http://127.0.0.1:46585/ using document root <none> and password file <none>
I20260928 04:58:09.409013 20851 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:09.409080 20851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:09.409225 20851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:09.410336 20851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "6ec72829fad548d98162ca01e02905f7"
format_stamp: "Formatted at 2026-09-28 04:58:09 on dist-test-slave-m39l"
server_key: "6bca3fba1a78495899d866669c2c00b9"
server_key_iv: "7694a58fae667b826a6a5445ca786d7c"
server_key_version: "encryptionkey@0"
I20260928 04:58:09.410759 20851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "6ec72829fad548d98162ca01e02905f7"
format_stamp: "Formatted at 2026-09-28 04:58:09 on dist-test-slave-m39l"
server_key: "6bca3fba1a78495899d866669c2c00b9"
server_key_iv: "7694a58fae667b826a6a5445ca786d7c"
server_key_version: "encryptionkey@0"
I20260928 04:58:09.412285 20851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:09.413165 20866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:09.413384 20851 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:09.413508 20851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "6ec72829fad548d98162ca01e02905f7"
format_stamp: "Formatted at 2026-09-28 04:58:09 on dist-test-slave-m39l"
server_key: "6bca3fba1a78495899d866669c2c00b9"
server_key_iv: "7694a58fae667b826a6a5445ca786d7c"
server_key_version: "encryptionkey@0"
I20260928 04:58:09.413617 20851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:09.417496 20851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:09.419765 20851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:09.419955 20851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:09.424039 20851 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37951
I20260928 04:58:09.424096 20918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37951 every 8 connection(s)
I20260928 04:58:09.424460 20851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:09.425527 20919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:09.427959 20919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7: Bootstrap starting.
I20260928 04:58:09.428524 20919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:09.428768 20919 log.cc:824] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:09.429564 20919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7: No bootstrap required, opened a new log
I20260928 04:58:09.431494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 20851
I20260928 04:58:09.431396 20919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec72829fad548d98162ca01e02905f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37951 } }
I20260928 04:58:09.431612 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:09.431603 20919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:09.431645 20919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ec72829fad548d98162ca01e02905f7, State: Initialized, Role: FOLLOWER
I20260928 04:58:09.431816 392 external_mini_cluster.cc:1428] Setting key 41e0159030526372b3f24c4cb6062a93
I20260928 04:58:09.431819 20919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec72829fad548d98162ca01e02905f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37951 } }
I20260928 04:58:09.431905 20919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:09.431948 20919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:09.432004 20919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:09.432821 20919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec72829fad548d98162ca01e02905f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37951 } }
I20260928 04:58:09.432962 20919 leader_election.cc:302] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ec72829fad548d98162ca01e02905f7; no voters:
I20260928 04:58:09.433127 20919 leader_election.cc:288] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:09.433182 20924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:09.433363 20924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7 [term 1 LEADER]: Becoming Leader. State: Replica: 6ec72829fad548d98162ca01e02905f7, State: Running, Role: LEADER
I20260928 04:58:09.433418 20919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:09.433458 20924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec72829fad548d98162ca01e02905f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37951 } }
I20260928 04:58:09.433873 20924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ec72829fad548d98162ca01e02905f7. Latest consensus state: current_term: 1 leader_uuid: "6ec72829fad548d98162ca01e02905f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec72829fad548d98162ca01e02905f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37951 } } }
I20260928 04:58:09.434105 20924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:09.434037 20925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ec72829fad548d98162ca01e02905f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec72829fad548d98162ca01e02905f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37951 } } }
I20260928 04:58:09.434345 20925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:09.435067 20923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:09.432425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58091 (local address 127.0.0.1:37951)
0928 04:58:09.432949 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:09.432960 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:09.432998 (+ 38us) server_negotiation.cc:407] Connection header received
0928 04:58:09.433132 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:09.433139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:09.433173 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:09.433868 (+ 695us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:09.434383 (+ 515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:09.434387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:09.434388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:09.434391 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:09.434431 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:09.434445 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:09.434728 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:09.434768 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 04:58:09.434819 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":395,"threads_started":1}
W20260928 04:58:09.435451 20940 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:58:09.435575 20940 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:58:09.435665 20933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:09.435851 20933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:09.436236 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:09.437583 20933 catalog_manager.cc:1347] Generated new cluster ID: 934006ab53ec44d69f171890a0940c25
I20260928 04:58:09.437634 20933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:58:09.438920 20923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:09.437775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57871 (local address 127.0.0.1:37951)
0928 04:58:09.437813 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:09.437815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:09.437927 (+ 112us) server_negotiation.cc:407] Connection header received
0928 04:58:09.438003 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:09.438004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:09.438009 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:09.438061 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:09.438683 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:09.438684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:09.438684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:09.438689 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:09.438704 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:09.438708 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:09.438766 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:09.438820 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 04:58:09.438836 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:58:09.452167 20933 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:09.452386 20933 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:09.457659 20933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ec72829fad548d98162ca01e02905f7: Generated new TSK 0
I20260928 04:58:09.457878 20933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:09.459929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 20851
2026-09-28T04:58:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:09 dist-test-slave-m39l krb5kdc[20950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:09 dist-test-slave-m39l krb5kdc[20950](info): set up 2 sockets
Sep 28 04:58:09 dist-test-slave-m39l krb5kdc[20950](info): commencing operation
krb5kdc: starting...
W20260928 04:58:10.526779 392 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:10 dist-test-slave-m39l krb5kdc[20950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:58:10.662591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45189
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:10.750427 20963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:10.750797 20963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:10.750828 20963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:10.752367 20963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:10.752431 20963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:10.752449 20963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:10.752465 20963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:10.752480 20963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:10.754271 20963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:10.754611 20963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:10.754906 20963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:10.760551 20968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:10.760484 20969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:10.760500 20963 server_base.cc:1034] running on GCE node
W20260928 04:58:10.760484 20971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:10.761310 20963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:10.761669 20963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:10.762919 20963 hybrid_clock.cc:648] HybridClock initialized: now 1790571490762898 us; error 38 us; skew 500 ppm
Sep 28 04:58:10 dist-test-slave-m39l krb5kdc[20950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:10.765959 20963 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:58:10.766647 20963 webserver.cc:460] Webserver started at http://127.0.0.1:34305/ using document root <none> and password file <none>
I20260928 04:58:10.766906 20963 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:10.766990 20963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:10.767148 20963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:10.768378 20963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "9c09320366594b86b858ad9c324b4459"
format_stamp: "Formatted at 2026-09-28 04:58:10 on dist-test-slave-m39l"
server_key: "61ccb64ab5bc4f9d5aff0cec8c143eb8"
server_key_iv: "b7441144b7658d31e1feed05a01b0fd0"
server_key_version: "encryptionkey@0"
I20260928 04:58:10.768793 20963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "9c09320366594b86b858ad9c324b4459"
format_stamp: "Formatted at 2026-09-28 04:58:10 on dist-test-slave-m39l"
server_key: "61ccb64ab5bc4f9d5aff0cec8c143eb8"
server_key_iv: "b7441144b7658d31e1feed05a01b0fd0"
server_key_version: "encryptionkey@0"
I20260928 04:58:10.770413 20963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:58:10.771440 20978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:10.771631 20963 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:10.771754 20963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "9c09320366594b86b858ad9c324b4459"
format_stamp: "Formatted at 2026-09-28 04:58:10 on dist-test-slave-m39l"
server_key: "61ccb64ab5bc4f9d5aff0cec8c143eb8"
server_key_iv: "b7441144b7658d31e1feed05a01b0fd0"
server_key_version: "encryptionkey@0"
I20260928 04:58:10.771847 20963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:10.795406 20963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:10.797773 20963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:10.797992 20963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:10.802925 20963 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45189
I20260928 04:58:10.802990 21030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45189 every 8 connection(s)
I20260928 04:58:10.803382 20963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:10.804314 21031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:10.806747 21031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459: Bootstrap starting.
I20260928 04:58:10.807307 21031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:10.807554 21031 log.cc:824] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:10.808413 21031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459: No bootstrap required, opened a new log
I20260928 04:58:10.808416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 20963
I20260928 04:58:10.808532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:10.808735 392 external_mini_cluster.cc:1428] Setting key 4be69c609f9665b770d526c6a63e1492
I20260928 04:58:10.810843 21031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c09320366594b86b858ad9c324b4459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45189 } }
I20260928 04:58:10.810995 21031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:10.811020 21031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c09320366594b86b858ad9c324b4459, State: Initialized, Role: FOLLOWER
I20260928 04:58:10.811223 21031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c09320366594b86b858ad9c324b4459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45189 } }
I20260928 04:58:10.811314 21031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:10.811349 21031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:10.811403 21031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:10.812372 21031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c09320366594b86b858ad9c324b4459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45189 } }
I20260928 04:58:10.812487 21031 leader_election.cc:302] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c09320366594b86b858ad9c324b4459; no voters:
I20260928 04:58:10.812743 21031 leader_election.cc:288] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:10.812801 21036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:10.812947 21036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459 [term 1 LEADER]: Becoming Leader. State: Replica: 9c09320366594b86b858ad9c324b4459, State: Running, Role: LEADER
I20260928 04:58:10.813057 21031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:10.813091 21036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c09320366594b86b858ad9c324b4459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45189 } }
I20260928 04:58:10.813519 21037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c09320366594b86b858ad9c324b4459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c09320366594b86b858ad9c324b4459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45189 } } }
I20260928 04:58:10.813591 21037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:10.813740 21038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c09320366594b86b858ad9c324b4459. Latest consensus state: current_term: 1 leader_uuid: "9c09320366594b86b858ad9c324b4459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c09320366594b86b858ad9c324b4459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45189 } } }
I20260928 04:58:10.813802 21038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:10.814343 21041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:10.815052 21041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:10.817145 21041 catalog_manager.cc:1347] Generated new cluster ID: 046ee534394646bb9ecaa0dfeaf9a776
I20260928 04:58:10.817207 21041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:58:10 dist-test-slave-m39l krb5kdc[20950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 04:58:10.823773 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:10.809302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50613 (local address 127.0.0.1:45189)
0928 04:58:10.809505 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:10.809513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:10.809540 (+ 27us) server_negotiation.cc:407] Connection header received
0928 04:58:10.810831 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:10.810841 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:10.810907 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:10.811100 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:10.811951 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:10.813711 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:10.814870 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:10.815586 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:10.815702 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:10.819268 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:10.819274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:10.819274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:10.819290 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:10.821816 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:10.822317 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:10.822320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:10.822321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:10.822472 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:10.822726 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:10.822727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:10.822727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:10.823083 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:10.823251 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:10.823363 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 04:58:10.823520 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":119,"threads_started":1}
I20260928 04:58:10.824944 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:10.825428 21041 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:10.826152 21041 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:10.833341 21041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c09320366594b86b858ad9c324b4459: Generated new TSK 0
I20260928 04:58:10.833628 21041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:10.839212 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:10.831001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33583 (local address 127.0.0.1:45189)
0928 04:58:10.831061 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:10.831064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:10.831354 (+ 290us) server_negotiation.cc:407] Connection header received
0928 04:58:10.831452 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:10.831453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:10.831491 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:10.831574 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:10.832335 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:10.832921 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:10.834234 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:10.834583 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:10.834666 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:10.836127 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:10.836129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:10.836129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:10.836138 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:10.837655 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:10.838279 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:10.838281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:10.838281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:10.838352 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:10.838611 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:10.838612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:10.838612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:10.838747 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:10.838857 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:10.838992 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 04:58:10.839098 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 04:58:10.841957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 20963
2026-09-28T04:58:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:10 dist-test-slave-m39l krb5kdc[21062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:10 dist-test-slave-m39l krb5kdc[21062](info): set up 2 sockets
Sep 28 04:58:10 dist-test-slave-m39l krb5kdc[21062](info): commencing operation
krb5kdc: starting...
W20260928 04:58:11.924042 392 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:12 dist-test-slave-m39l krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:58:12.053984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:12.140724 21075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:12.141043 21075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:12.141093 21075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:12.142454 21075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:12.142572 21075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:12.142609 21075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:12.142627 21075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:12.142642 21075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:12.144284 21075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:12.144518 21075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:12.144721 21075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:12.150565 21083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:12.150561 21081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:12.150635 21080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:12.151050 21075 server_base.cc:1034] running on GCE node
I20260928 04:58:12.151293 21075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:12.151623 21075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:12.152832 21075 hybrid_clock.cc:648] HybridClock initialized: now 1790571492152800 us; error 38 us; skew 500 ppm
Sep 28 04:58:12 dist-test-slave-m39l krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:12.162365 21075 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:58:12.162995 21075 webserver.cc:460] Webserver started at http://127.0.0.1:37073/ using document root <none> and password file <none>
I20260928 04:58:12.163309 21075 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:12.163383 21075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:12.163543 21075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:12.164661 21075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "941df1d71b054a64973b1bb6e017ff93"
format_stamp: "Formatted at 2026-09-28 04:58:12 on dist-test-slave-m39l"
server_key: "21e6076e525810840913246bd83bd2b4"
server_key_iv: "a7325f6a51c5290fd8b33810d824906e"
server_key_version: "encryptionkey@0"
I20260928 04:58:12.165037 21075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "941df1d71b054a64973b1bb6e017ff93"
format_stamp: "Formatted at 2026-09-28 04:58:12 on dist-test-slave-m39l"
server_key: "21e6076e525810840913246bd83bd2b4"
server_key_iv: "a7325f6a51c5290fd8b33810d824906e"
server_key_version: "encryptionkey@0"
I20260928 04:58:12.166615 21075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:58:12.167447 21090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:12.167636 21075 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:58:12.167742 21075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "941df1d71b054a64973b1bb6e017ff93"
format_stamp: "Formatted at 2026-09-28 04:58:12 on dist-test-slave-m39l"
server_key: "21e6076e525810840913246bd83bd2b4"
server_key_iv: "a7325f6a51c5290fd8b33810d824906e"
server_key_version: "encryptionkey@0"
I20260928 04:58:12.167829 21075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:12.187849 21075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:12.189908 21075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:12.190081 21075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:12.194334 21075 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46593
I20260928 04:58:12.194370 21142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46593 every 8 connection(s)
I20260928 04:58:12.194813 21075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:12.195698 21143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:12.197980 21143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93: Bootstrap starting.
I20260928 04:58:12.198643 21143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:12.198885 21143 log.cc:824] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:12.199565 21143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93: No bootstrap required, opened a new log
I20260928 04:58:12.200260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 21075
I20260928 04:58:12.200397 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:12.200575 392 external_mini_cluster.cc:1428] Setting key 0bcc2d4478723aae23390e41f211f89e
I20260928 04:58:12.201749 21143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941df1d71b054a64973b1bb6e017ff93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260928 04:58:12.201910 21143 raft_consensus.cc:374] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:12.201948 21143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 941df1d71b054a64973b1bb6e017ff93, State: Initialized, Role: FOLLOWER
I20260928 04:58:12.202137 21143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941df1d71b054a64973b1bb6e017ff93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260928 04:58:12.202227 21143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:12.202251 21143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:12.202306 21143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:12.202926 21143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941df1d71b054a64973b1bb6e017ff93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260928 04:58:12.203017 21143 leader_election.cc:302] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 941df1d71b054a64973b1bb6e017ff93; no voters:
I20260928 04:58:12.203189 21143 leader_election.cc:288] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:12.203217 21148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:12.203316 21148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93 [term 1 LEADER]: Becoming Leader. State: Replica: 941df1d71b054a64973b1bb6e017ff93, State: Running, Role: LEADER
I20260928 04:58:12.203446 21143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:12.203444 21148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941df1d71b054a64973b1bb6e017ff93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260928 04:58:12.204041 21149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "941df1d71b054a64973b1bb6e017ff93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941df1d71b054a64973b1bb6e017ff93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } } }
I20260928 04:58:12.204061 21150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 941df1d71b054a64973b1bb6e017ff93. Latest consensus state: current_term: 1 leader_uuid: "941df1d71b054a64973b1bb6e017ff93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941df1d71b054a64973b1bb6e017ff93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } } }
I20260928 04:58:12.204213 21149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:12.204474 21150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:12.204658 21157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:12.205551 21157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:12.207216 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:12.201143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42127 (local address 127.0.0.1:46593)
0928 04:58:12.201339 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:12.201348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:12.201378 (+ 30us) server_negotiation.cc:407] Connection header received
0928 04:58:12.201731 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:12.201738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:12.201796 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:12.202008 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:12.202153 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:12.203870 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:12.205141 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:12.205563 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:12.205684 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:12.206436 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:12.206441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:12.206441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:12.206732 (+ 291us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:12.206772 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:12.206791 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:12.206881 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:12.206957 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 04:58:12.207031 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260928 04:58:12.208141 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:12.210950 21157 catalog_manager.cc:1347] Generated new cluster ID: 0374aa8f6d9841b5813dadc3d3fcdc13
I20260928 04:58:12.211030 21157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:58:12.212997 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:12.209365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47109 (local address 127.0.0.1:46593)
0928 04:58:12.209401 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:12.209402 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:12.209532 (+ 130us) server_negotiation.cc:407] Connection header received
0928 04:58:12.209598 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:12.209599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:12.209607 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:12.209653 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:12.209806 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:12.210255 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:12.211450 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:12.211777 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:12.211823 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:12.212638 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:12.212639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:12.212640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:12.212646 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:12.212662 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:12.212667 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:12.212706 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:12.212834 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 04:58:12.212867 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 04:58:12.240468 21157 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:12.241426 21157 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:12.249634 21157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 941df1d71b054a64973b1bb6e017ff93: Generated new TSK 0
I20260928 04:58:12.249914 21157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:12.253626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 21075
2026-09-28T04:58:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:12 dist-test-slave-m39l krb5kdc[21174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:12 dist-test-slave-m39l krb5kdc[21174](info): set up 2 sockets
Sep 28 04:58:12 dist-test-slave-m39l krb5kdc[21174](info): commencing operation
krb5kdc: starting...
W20260928 04:58:13.311144 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:13 dist-test-slave-m39l krb5kdc[21174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:58:13.445451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:13.529804 21187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:13.530105 21187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:13.530151 21187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:13.531528 21187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:13.531617 21187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:13.531643 21187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:13.531665 21187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:13.531695 21187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:13.533331 21187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:13.533648 21187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:13.533906 21187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:13.539570 21193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:13.539593 21195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:13.539570 21192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:13.539876 21187 server_base.cc:1034] running on GCE node
I20260928 04:58:13.540145 21187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:13.540467 21187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:13.541713 21187 hybrid_clock.cc:648] HybridClock initialized: now 1790571493541700 us; error 53 us; skew 500 ppm
Sep 28 04:58:13 dist-test-slave-m39l krb5kdc[21174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:13.544797 21187 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:58:13.545418 21187 webserver.cc:460] Webserver started at http://127.0.0.1:34025/ using document root <none> and password file <none>
I20260928 04:58:13.545675 21187 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:13.545751 21187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:13.545910 21187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:13.547209 21187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "c5f5af5e03c440d788060cd157eb6cec"
format_stamp: "Formatted at 2026-09-28 04:58:13 on dist-test-slave-m39l"
server_key: "34cb1b86acc5cc1e1de0dee62a6a9b84"
server_key_iv: "48763da30ded980278d7328e4820ca0d"
server_key_version: "encryptionkey@0"
I20260928 04:58:13.547616 21187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "c5f5af5e03c440d788060cd157eb6cec"
format_stamp: "Formatted at 2026-09-28 04:58:13 on dist-test-slave-m39l"
server_key: "34cb1b86acc5cc1e1de0dee62a6a9b84"
server_key_iv: "48763da30ded980278d7328e4820ca0d"
server_key_version: "encryptionkey@0"
I20260928 04:58:13.549270 21187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:58:13.550261 21202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:13.550616 21187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:13.550752 21187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "c5f5af5e03c440d788060cd157eb6cec"
format_stamp: "Formatted at 2026-09-28 04:58:13 on dist-test-slave-m39l"
server_key: "34cb1b86acc5cc1e1de0dee62a6a9b84"
server_key_iv: "48763da30ded980278d7328e4820ca0d"
server_key_version: "encryptionkey@0"
I20260928 04:58:13.550859 21187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:13.554832 21187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:13.557535 21187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:13.557763 21187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:13.562724 21187 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42583
I20260928 04:58:13.562763 21254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42583 every 8 connection(s)
I20260928 04:58:13.563198 21187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:13.564306 21255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:13.566845 21255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec: Bootstrap starting.
I20260928 04:58:13.567484 21255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:13.567780 21255 log.cc:824] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:13.568643 21255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec: No bootstrap required, opened a new log
I20260928 04:58:13.570354 21255 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f5af5e03c440d788060cd157eb6cec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42583 } }
I20260928 04:58:13.570586 21255 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:13.570632 21255 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5f5af5e03c440d788060cd157eb6cec, State: Initialized, Role: FOLLOWER
I20260928 04:58:13.570729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 21187
I20260928 04:58:13.570789 21255 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f5af5e03c440d788060cd157eb6cec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42583 } }
I20260928 04:58:13.570881 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:13.570895 21255 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:13.571017 21255 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:13.571076 21255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:13.571221 392 external_mini_cluster.cc:1428] Setting key 1ee131ac86efe63437caf4cc0040b1ae
I20260928 04:58:13.571965 21255 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f5af5e03c440d788060cd157eb6cec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42583 } }
I20260928 04:58:13.572099 21255 leader_election.cc:302] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5f5af5e03c440d788060cd157eb6cec; no voters:
I20260928 04:58:13.572312 21255 leader_election.cc:288] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:13.572345 21259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:13.572467 21259 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec [term 1 LEADER]: Becoming Leader. State: Replica: c5f5af5e03c440d788060cd157eb6cec, State: Running, Role: LEADER
I20260928 04:58:13.572655 21255 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:13.572620 21259 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f5af5e03c440d788060cd157eb6cec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42583 } }
I20260928 04:58:13.573180 21261 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5f5af5e03c440d788060cd157eb6cec. Latest consensus state: current_term: 1 leader_uuid: "c5f5af5e03c440d788060cd157eb6cec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f5af5e03c440d788060cd157eb6cec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42583 } } }
I20260928 04:58:13.573264 21261 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:13.573455 21262 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5f5af5e03c440d788060cd157eb6cec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f5af5e03c440d788060cd157eb6cec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42583 } } }
I20260928 04:58:13.573604 21262 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:13.573843 21265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:13.574442 21265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:13.574539 21260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:13.572299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32957 (local address 127.0.0.1:42583)
0928 04:58:13.572530 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:13.572542 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:13.572580 (+ 38us) server_negotiation.cc:407] Connection header received
0928 04:58:13.572718 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:13.572726 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:13.572760 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:13.573228 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:13.573803 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:13.573806 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:13.573807 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:13.573811 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:13.573856 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:13.573874 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:13.574214 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:13.574252 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 04:58:13.574302 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":130,"threads_started":1}
I20260928 04:58:13.575738 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:13.576261 21265 catalog_manager.cc:1347] Generated new cluster ID: dc74adf65df94ac6965d680701606b46
I20260928 04:58:13.576325 21265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:58:13.579813 21260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:13.577205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45567 (local address 127.0.0.1:42583)
0928 04:58:13.577244 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:13.577246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:13.578769 (+ 1523us) server_negotiation.cc:407] Connection header received
0928 04:58:13.578844 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:13.578845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:13.578853 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:13.578911 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:13.579574 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:13.579576 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:13.579576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:13.579580 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:13.579598 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:13.579603 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:13.579653 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:13.579705 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 04:58:13.579721 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:58:13.590598 21265 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:13.590833 21265 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:13.605762 21265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5f5af5e03c440d788060cd157eb6cec: Generated new TSK 0
I20260928 04:58:13.606101 21265 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:13.609614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 21187
2026-09-28T04:58:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:13 dist-test-slave-m39l krb5kdc[21286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:13 dist-test-slave-m39l krb5kdc[21286](info): set up 2 sockets
Sep 28 04:58:13 dist-test-slave-m39l krb5kdc[21286](info): commencing operation
krb5kdc: starting...
W20260928 04:58:14.671217 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:14 dist-test-slave-m39l krb5kdc[21286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:58:14.803187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:14.890234 21299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:14.890524 21299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:14.890650 21299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:14.891898 21299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:14.891988 21299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:14.892019 21299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:14.892050 21299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:14.892077 21299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:14.893631 21299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:14.893937 21299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:14.894191 21299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:14.899910 21305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:14.900023 21304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:14.900054 21307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:14.900295 21299 server_base.cc:1034] running on GCE node
I20260928 04:58:14.900543 21299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:14.900851 21299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:14.902051 21299 hybrid_clock.cc:648] HybridClock initialized: now 1790571494902025 us; error 37 us; skew 500 ppm
Sep 28 04:58:14 dist-test-slave-m39l krb5kdc[21286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:14.904796 21299 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:58:14.905314 21299 webserver.cc:460] Webserver started at http://127.0.0.1:39535/ using document root <none> and password file <none>
I20260928 04:58:14.905550 21299 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:14.905617 21299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:14.905778 21299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:14.906978 21299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "12b0eba9303043a4ae1b868828985148"
format_stamp: "Formatted at 2026-09-28 04:58:14 on dist-test-slave-m39l"
server_key: "8ddf68f87ef2cd3e1988f77b06fa95d6"
server_key_iv: "0d582150f8977aaf5350a4ee20d1890e"
server_key_version: "encryptionkey@0"
I20260928 04:58:14.907341 21299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "12b0eba9303043a4ae1b868828985148"
format_stamp: "Formatted at 2026-09-28 04:58:14 on dist-test-slave-m39l"
server_key: "8ddf68f87ef2cd3e1988f77b06fa95d6"
server_key_iv: "0d582150f8977aaf5350a4ee20d1890e"
server_key_version: "encryptionkey@0"
I20260928 04:58:14.908829 21299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:58:14.909746 21314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:14.909998 21299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:14.910127 21299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "12b0eba9303043a4ae1b868828985148"
format_stamp: "Formatted at 2026-09-28 04:58:14 on dist-test-slave-m39l"
server_key: "8ddf68f87ef2cd3e1988f77b06fa95d6"
server_key_iv: "0d582150f8977aaf5350a4ee20d1890e"
server_key_version: "encryptionkey@0"
I20260928 04:58:14.910219 21299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:14.920368 21299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:14.923045 21299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:14.923245 21299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:14.927994 21299 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34127
I20260928 04:58:14.928030 21366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34127 every 8 connection(s)
I20260928 04:58:14.928457 21299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:14.929471 21367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:14.932310 21367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148: Bootstrap starting.
I20260928 04:58:14.932996 21367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:14.933315 21367 log.cc:824] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:14.934141 21367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148: No bootstrap required, opened a new log
I20260928 04:58:14.935951 21367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b0eba9303043a4ae1b868828985148" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34127 } }
I20260928 04:58:14.936102 21367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:14.936124 21367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b0eba9303043a4ae1b868828985148, State: Initialized, Role: FOLLOWER
I20260928 04:58:14.936291 21367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b0eba9303043a4ae1b868828985148" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34127 } }
I20260928 04:58:14.936358 21367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:14.936379 21367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:14.936460 21367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:14.937059 21367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b0eba9303043a4ae1b868828985148" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34127 } }
I20260928 04:58:14.937171 21367 leader_election.cc:302] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12b0eba9303043a4ae1b868828985148; no voters:
I20260928 04:58:14.937371 21367 leader_election.cc:288] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:14.937500 21370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:14.937659 21367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:14.937732 21370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148 [term 1 LEADER]: Becoming Leader. State: Replica: 12b0eba9303043a4ae1b868828985148, State: Running, Role: LEADER
I20260928 04:58:14.937911 21370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b0eba9303043a4ae1b868828985148" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34127 } }
I20260928 04:58:14.938768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 21299
I20260928 04:58:14.938913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:14.939126 392 external_mini_cluster.cc:1428] Setting key a7f542d254d8e71433a2dd512cd0bffc
I20260928 04:58:14.939560 21372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12b0eba9303043a4ae1b868828985148. Latest consensus state: current_term: 1 leader_uuid: "12b0eba9303043a4ae1b868828985148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b0eba9303043a4ae1b868828985148" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34127 } } }
I20260928 04:58:14.939953 21372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148 [sys.catalog]: This master's current role is: LEADER
W20260928 04:58:14.940006 21384 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:58:14.940052 21384 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:58:14.939607 21371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12b0eba9303043a4ae1b868828985148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b0eba9303043a4ae1b868828985148" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34127 } } }
I20260928 04:58:14.940169 21371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:14.940325 21389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:14.940539 21389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:14.942296 21389 catalog_manager.cc:1347] Generated new cluster ID: 37c3f74544e74c79861517c50c5c86c9
I20260928 04:58:14.942358 21389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:58:14 dist-test-slave-m39l krb5kdc[21286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 04:58:14.949646 21389 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:14.950572 21389 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:14.953765 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:14.939879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:40687 (local address 127.0.0.1:34127)
0928 04:58:14.940055 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:14.940069 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:14.940106 (+ 37us) server_negotiation.cc:407] Connection header received
0928 04:58:14.940270 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:14.940279 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:14.940354 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:14.940520 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:14.941290 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:14.942816 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:14.944024 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:14.944758 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:14.944871 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:14.948234 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:14.948241 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:14.948242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:14.948259 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:14.951816 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:14.952390 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:14.952395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:14.952395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:14.952537 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:14.952792 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:14.952793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:14.952794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:14.953110 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:14.953299 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:14.953383 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 04:58:14.953492 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260928 04:58:14.954876 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:14.960093 21389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12b0eba9303043a4ae1b868828985148: Generated new TSK 0
I20260928 04:58:14.960317 21389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:14.966847 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:14.958898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:55639 (local address 127.0.0.1:34127)
0928 04:58:14.958952 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:14.958955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:14.959164 (+ 209us) server_negotiation.cc:407] Connection header received
0928 04:58:14.959228 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:14.959229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:14.959241 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:14.959299 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:14.960049 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:14.961329 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:14.962131 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:14.962476 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:14.962582 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:14.963878 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:14.963880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:14.963880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:14.963888 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:14.965580 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:14.965968 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:14.965969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:14.965969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:14.966032 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:14.966255 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:14.966256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:14.966256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:14.966391 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:14.966558 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:14.966680 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 04:58:14.966748 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:58:14.969264 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 21299
W20260928 04:58:14.974133 21284 connection.cc:535] client connection to 127.0.0.1:34127 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34127): Connection reset by peer (error 104)
2026-09-28T04:58:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:15 dist-test-slave-m39l krb5kdc[21398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:15 dist-test-slave-m39l krb5kdc[21398](info): set up 2 sockets
Sep 28 04:58:15 dist-test-slave-m39l krb5kdc[21398](info): commencing operation
krb5kdc: starting...
W20260928 04:58:16.043113 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:16 dist-test-slave-m39l krb5kdc[21398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:58:16.178380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:16.268462 21411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:16.268760 21411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:16.268810 21411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:16.270635 21411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:16.270723 21411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:16.270751 21411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:16.270788 21411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:16.270820 21411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:16.272710 21411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:16.273074 21411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:16.273350 21411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:16.279402 21419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:16.279413 21417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:16.279397 21416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:16.279439 21411 server_base.cc:1034] running on GCE node
I20260928 04:58:16.279873 21411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:16.280241 21411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:16.281467 21411 hybrid_clock.cc:648] HybridClock initialized: now 1790571496281402 us; error 72 us; skew 500 ppm
Sep 28 04:58:16 dist-test-slave-m39l krb5kdc[21398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:16.284492 21411 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:58:16.285037 21411 webserver.cc:460] Webserver started at http://127.0.0.1:36809/ using document root <none> and password file <none>
I20260928 04:58:16.285318 21411 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:16.285399 21411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:16.285571 21411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:16.287357 21411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "3e41446638ee4cae9bc9a41caddc4ab2"
format_stamp: "Formatted at 2026-09-28 04:58:16 on dist-test-slave-m39l"
server_key: "9e69ee168f2b22501a20a0c128dc9cca"
server_key_iv: "3e54fbcbbfaeb7d43d1e6ef0e1203cfc"
server_key_version: "encryptionkey@0"
I20260928 04:58:16.287783 21411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "3e41446638ee4cae9bc9a41caddc4ab2"
format_stamp: "Formatted at 2026-09-28 04:58:16 on dist-test-slave-m39l"
server_key: "9e69ee168f2b22501a20a0c128dc9cca"
server_key_iv: "3e54fbcbbfaeb7d43d1e6ef0e1203cfc"
server_key_version: "encryptionkey@0"
I20260928 04:58:16.289777 21411 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 04:58:16.290745 21426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:16.291059 21411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:16.291195 21411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "3e41446638ee4cae9bc9a41caddc4ab2"
format_stamp: "Formatted at 2026-09-28 04:58:16 on dist-test-slave-m39l"
server_key: "9e69ee168f2b22501a20a0c128dc9cca"
server_key_iv: "3e54fbcbbfaeb7d43d1e6ef0e1203cfc"
server_key_version: "encryptionkey@0"
I20260928 04:58:16.291296 21411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:16.319922 21411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:16.322257 21411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:16.322566 21411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:16.327189 21411 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46317
I20260928 04:58:16.327239 21478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46317 every 8 connection(s)
I20260928 04:58:16.327662 21411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:16.328621 21479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:16.331266 21479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2: Bootstrap starting.
I20260928 04:58:16.332010 21479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:16.332319 21479 log.cc:824] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:16.333127 21479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2: No bootstrap required, opened a new log
I20260928 04:58:16.334044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 21411
I20260928 04:58:16.334178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:16.334371 392 external_mini_cluster.cc:1428] Setting key b443c43ca501087a300a8aeb02f6b6e0
I20260928 04:58:16.335633 21479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e41446638ee4cae9bc9a41caddc4ab2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46317 } }
I20260928 04:58:16.335836 21479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:16.335876 21479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e41446638ee4cae9bc9a41caddc4ab2, State: Initialized, Role: FOLLOWER
I20260928 04:58:16.336025 21479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e41446638ee4cae9bc9a41caddc4ab2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46317 } }
I20260928 04:58:16.336118 21479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:16.336180 21479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:16.336226 21479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:16.336877 21479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e41446638ee4cae9bc9a41caddc4ab2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46317 } }
I20260928 04:58:16.336971 21479 leader_election.cc:302] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e41446638ee4cae9bc9a41caddc4ab2; no voters:
I20260928 04:58:16.337204 21479 leader_election.cc:288] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:16.337229 21484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:16.337345 21484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2 [term 1 LEADER]: Becoming Leader. State: Replica: 3e41446638ee4cae9bc9a41caddc4ab2, State: Running, Role: LEADER
I20260928 04:58:16.337507 21479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:16.337496 21484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e41446638ee4cae9bc9a41caddc4ab2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46317 } }
I20260928 04:58:16.338032 21485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e41446638ee4cae9bc9a41caddc4ab2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e41446638ee4cae9bc9a41caddc4ab2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46317 } } }
I20260928 04:58:16.338291 21485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:16.338212 21486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e41446638ee4cae9bc9a41caddc4ab2. Latest consensus state: current_term: 1 leader_uuid: "3e41446638ee4cae9bc9a41caddc4ab2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e41446638ee4cae9bc9a41caddc4ab2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46317 } } }
I20260928 04:58:16.338630 21486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:16.338796 21493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:16.339478 21493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:16.341665 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:16.335150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:36837 (local address 127.0.0.1:46317)
0928 04:58:16.335377 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:16.335389 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:16.335432 (+ 43us) server_negotiation.cc:407] Connection header received
0928 04:58:16.335568 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:16.335576 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:16.335639 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:16.335796 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:16.336101 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:16.337781 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:16.339181 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:16.339793 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:16.339947 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:16.340710 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:16.340715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:16.340715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:16.341005 (+ 290us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:16.341049 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:16.341068 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:16.341171 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:16.341331 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 04:58:16.341366 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":101,"threads_started":1}
I20260928 04:58:16.342610 21493 catalog_manager.cc:1347] Generated new cluster ID: 518f7b18abe94b3c89bd3f1748b3f6c2
I20260928 04:58:16.342675 21493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:58:16.342880 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:16.349010 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:16.344522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:43921 (local address 127.0.0.1:46317)
0928 04:58:16.344615 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:16.344617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:16.345054 (+ 437us) server_negotiation.cc:407] Connection header received
0928 04:58:16.345127 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:16.345128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:16.345143 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:16.345208 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:16.345395 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:16.346217 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:16.347413 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:16.347774 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:16.347838 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:16.348718 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:16.348720 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:16.348720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:16.348726 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:16.348743 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:16.348748 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:16.348810 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:16.348895 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 04:58:16.348909 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 04:58:16.372342 21493 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:16.373133 21493 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:16.385923 21493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e41446638ee4cae9bc9a41caddc4ab2: Generated new TSK 0
I20260928 04:58:16.386214 21493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:16.398479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 21411
W20260928 04:58:16.403461 21396 connection.cc:535] client connection to 127.0.0.1:46317 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46317): Connection reset by peer (error 104)
2026-09-28T04:58:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:16 dist-test-slave-m39l krb5kdc[21510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:16 dist-test-slave-m39l krb5kdc[21510](info): set up 2 sockets
Sep 28 04:58:16 dist-test-slave-m39l krb5kdc[21510](info): commencing operation
krb5kdc: starting...
W20260928 04:58:17.455132 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:17 dist-test-slave-m39l krb5kdc[21510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:58:17.590263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:17.675719 21523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:17.676091 21523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:17.676139 21523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:17.677376 21523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:17.677457 21523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:17.677492 21523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:17.677520 21523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:17.677552 21523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:17.679137 21523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:17.679451 21523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:17.679709 21523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:17.685341 21529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:17.685341 21528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:17.685367 21531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:17.685621 21523 server_base.cc:1034] running on GCE node
I20260928 04:58:17.685909 21523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:17.686285 21523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:17.687479 21523 hybrid_clock.cc:648] HybridClock initialized: now 1790571497687435 us; error 55 us; skew 500 ppm
Sep 28 04:58:17 dist-test-slave-m39l krb5kdc[21510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:17.696058 21523 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:58:17.696745 21523 webserver.cc:460] Webserver started at http://127.0.0.1:34359/ using document root <none> and password file <none>
I20260928 04:58:17.697032 21523 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:17.697120 21523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:17.697432 21523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:17.699467 21523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "fb2ac4b1097041258a29e2fcc70088f5"
format_stamp: "Formatted at 2026-09-28 04:58:17 on dist-test-slave-m39l"
server_key: "939c764a593ab0dabd99553ab02d3385"
server_key_iv: "3183ab707d33503fb3289c22b6cb8636"
server_key_version: "encryptionkey@0"
I20260928 04:58:17.699955 21523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb2ac4b1097041258a29e2fcc70088f5"
format_stamp: "Formatted at 2026-09-28 04:58:17 on dist-test-slave-m39l"
server_key: "939c764a593ab0dabd99553ab02d3385"
server_key_iv: "3183ab707d33503fb3289c22b6cb8636"
server_key_version: "encryptionkey@0"
I20260928 04:58:17.701833 21523 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 04:58:17.703740 21538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:17.704190 21523 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:58:17.704348 21523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "fb2ac4b1097041258a29e2fcc70088f5"
format_stamp: "Formatted at 2026-09-28 04:58:17 on dist-test-slave-m39l"
server_key: "939c764a593ab0dabd99553ab02d3385"
server_key_iv: "3183ab707d33503fb3289c22b6cb8636"
server_key_version: "encryptionkey@0"
I20260928 04:58:17.704588 21523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:17.709434 21523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:17.713315 21523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:17.713752 21523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:17.720695 21523 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40315
I20260928 04:58:17.720774 21590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40315 every 8 connection(s)
I20260928 04:58:17.721360 21523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:17.722779 21591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:17.725411 21591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5: Bootstrap starting.
I20260928 04:58:17.725895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 21523
I20260928 04:58:17.726027 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:17.726222 392 external_mini_cluster.cc:1428] Setting key b9b65c6073109af097b37f109a0719af
I20260928 04:58:17.726267 21591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:17.726670 21591 log.cc:824] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:17.727592 21591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5: No bootstrap required, opened a new log
I20260928 04:58:17.730077 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:17.726970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:39277 (local address 127.0.0.1:40315)
0928 04:58:17.727225 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:17.727236 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:17.727273 (+ 37us) server_negotiation.cc:407] Connection header received
0928 04:58:17.728117 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:17.728129 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:17.728184 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:17.728406 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:17.729473 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:17.729477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:17.729478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:17.729483 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:17.729535 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:17.729551 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:17.729658 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:17.729765 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 04:58:17.729818 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
I20260928 04:58:17.731029 21591 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2ac4b1097041258a29e2fcc70088f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40315 } }
I20260928 04:58:17.731258 21591 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:17.731303 21591 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb2ac4b1097041258a29e2fcc70088f5, State: Initialized, Role: FOLLOWER
I20260928 04:58:17.731446 21591 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2ac4b1097041258a29e2fcc70088f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40315 } }
I20260928 04:58:17.731557 21591 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:17.731604 21591 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:17.731477 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:17.731657 21591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:17.732446 21591 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2ac4b1097041258a29e2fcc70088f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40315 } }
I20260928 04:58:17.732586 21591 leader_election.cc:302] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb2ac4b1097041258a29e2fcc70088f5; no voters:
I20260928 04:58:17.732826 21591 leader_election.cc:288] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:17.732944 21596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:17.733117 21596 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5 [term 1 LEADER]: Becoming Leader. State: Replica: fb2ac4b1097041258a29e2fcc70088f5, State: Running, Role: LEADER
I20260928 04:58:17.733122 21591 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:17.733232 21596 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2ac4b1097041258a29e2fcc70088f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40315 } }
I20260928 04:58:17.733587 21596 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb2ac4b1097041258a29e2fcc70088f5. Latest consensus state: current_term: 1 leader_uuid: "fb2ac4b1097041258a29e2fcc70088f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2ac4b1097041258a29e2fcc70088f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40315 } } }
I20260928 04:58:17.733673 21596 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:17.733630 21597 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb2ac4b1097041258a29e2fcc70088f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2ac4b1097041258a29e2fcc70088f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40315 } } }
I20260928 04:58:17.733896 21597 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:17.733917 21608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:17.734611 21608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:17.736660 21608 catalog_manager.cc:1347] Generated new cluster ID: 82a1fd5cbfd14fc9a5de1f90c570a06e
I20260928 04:58:17.736725 21608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:58:17.737363 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:17.735847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:42427 (local address 127.0.0.1:40315)
0928 04:58:17.735892 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:17.735894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:17.736162 (+ 268us) server_negotiation.cc:407] Connection header received
0928 04:58:17.736248 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:17.736249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:17.736256 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:17.736314 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:17.737106 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:17.737108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:17.737108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:17.737112 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:17.737128 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:17.737132 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:17.737190 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:17.737285 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 04:58:17.737299 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:58:17.743571 21608 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:17.743785 21608 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:17.746989 21608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb2ac4b1097041258a29e2fcc70088f5: Generated new TSK 0
I20260928 04:58:17.747213 21608 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:17.749212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 21523
2026-09-28T04:58:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:17 dist-test-slave-m39l krb5kdc[21622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:17 dist-test-slave-m39l krb5kdc[21622](info): set up 2 sockets
Sep 28 04:58:17 dist-test-slave-m39l krb5kdc[21622](info): commencing operation
krb5kdc: starting...
W20260928 04:58:18.791055 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:18 dist-test-slave-m39l krb5kdc[21622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:58:18.922250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38559
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:19.007050 21635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:19.007344 21635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:19.007390 21635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:19.008726 21635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:19.008816 21635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:19.008841 21635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:19.008864 21635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:19.008893 21635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:19.010665 21635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38559
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:19.011010 21635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:19.011270 21635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:19.016608 21643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:19.016574 21640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:19.016598 21641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:19.017198 21635 server_base.cc:1034] running on GCE node
I20260928 04:58:19.017437 21635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:19.017728 21635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:19.018908 21635 hybrid_clock.cc:648] HybridClock initialized: now 1790571499018888 us; error 34 us; skew 500 ppm
Sep 28 04:58:19 dist-test-slave-m39l krb5kdc[21622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:19.022024 21635 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:58:19.022835 21635 webserver.cc:460] Webserver started at http://127.0.0.1:40833/ using document root <none> and password file <none>
I20260928 04:58:19.023098 21635 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:19.023168 21635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:19.023321 21635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:19.024437 21635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "52ec7ddf82584de4a8eca0466b327c42"
format_stamp: "Formatted at 2026-09-28 04:58:19 on dist-test-slave-m39l"
server_key: "60f6730407de8bd7910bf8ade9700b96"
server_key_iv: "5e8868119501388165113f110cec9ff1"
server_key_version: "encryptionkey@0"
I20260928 04:58:19.024811 21635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "52ec7ddf82584de4a8eca0466b327c42"
format_stamp: "Formatted at 2026-09-28 04:58:19 on dist-test-slave-m39l"
server_key: "60f6730407de8bd7910bf8ade9700b96"
server_key_iv: "5e8868119501388165113f110cec9ff1"
server_key_version: "encryptionkey@0"
I20260928 04:58:19.026297 21635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:58:19.027316 21650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:19.027524 21635 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:19.027629 21635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "52ec7ddf82584de4a8eca0466b327c42"
format_stamp: "Formatted at 2026-09-28 04:58:19 on dist-test-slave-m39l"
server_key: "60f6730407de8bd7910bf8ade9700b96"
server_key_iv: "5e8868119501388165113f110cec9ff1"
server_key_version: "encryptionkey@0"
I20260928 04:58:19.027700 21635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:19.036574 21635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:19.039240 21635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:19.039366 21635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:19.043610 21635 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38559
I20260928 04:58:19.043642 21702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38559 every 8 connection(s)
I20260928 04:58:19.044023 21635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:19.045060 21703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:19.047577 21703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42: Bootstrap starting.
I20260928 04:58:19.047823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 21635
I20260928 04:58:19.047998 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:19.048194 392 external_mini_cluster.cc:1428] Setting key 4adc592e2df4a1fdbb21d287c35a21bc
I20260928 04:58:19.048331 21703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:19.048710 21703 log.cc:824] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:19.049537 21703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42: No bootstrap required, opened a new log
I20260928 04:58:19.051983 21703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ec7ddf82584de4a8eca0466b327c42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38559 } }
I20260928 04:58:19.052186 21703 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:19.052224 21703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ec7ddf82584de4a8eca0466b327c42, State: Initialized, Role: FOLLOWER
I20260928 04:58:19.052352 21703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ec7ddf82584de4a8eca0466b327c42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38559 } }
I20260928 04:58:19.052426 21703 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:19.052455 21703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:19.052500 21703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:19.053184 21703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ec7ddf82584de4a8eca0466b327c42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38559 } }
I20260928 04:58:19.053325 21703 leader_election.cc:302] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52ec7ddf82584de4a8eca0466b327c42; no voters:
I20260928 04:58:19.053521 21703 leader_election.cc:288] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:19.053627 21708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:19.053794 21708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42 [term 1 LEADER]: Becoming Leader. State: Replica: 52ec7ddf82584de4a8eca0466b327c42, State: Running, Role: LEADER
I20260928 04:58:19.053838 21703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:19.053892 21708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ec7ddf82584de4a8eca0466b327c42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38559 } }
I20260928 04:58:19.054224 21708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52ec7ddf82584de4a8eca0466b327c42. Latest consensus state: current_term: 1 leader_uuid: "52ec7ddf82584de4a8eca0466b327c42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ec7ddf82584de4a8eca0466b327c42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38559 } } }
I20260928 04:58:19.054230 21709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52ec7ddf82584de4a8eca0466b327c42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ec7ddf82584de4a8eca0466b327c42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38559 } } }
I20260928 04:58:19.054318 21708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:19.054322 21709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:19.054593 21715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:19.055095 21715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 04:58:19 dist-test-slave-m39l krb5kdc[21622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 04:58:19.056931 21715 catalog_manager.cc:1347] Generated new cluster ID: c7087657bb9347b9b1822c020e8d759b
I20260928 04:58:19.057003 21715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:58:19.060846 21707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:19.048847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:50739 (local address 127.0.0.1:38559)
0928 04:58:19.049051 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:19.049060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:19.049098 (+ 38us) server_negotiation.cc:407] Connection header received
0928 04:58:19.049584 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:19.049592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:19.049675 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:19.049827 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:19.050671 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:19.051643 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:19.052705 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:19.053187 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:19.053303 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:19.056897 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:19.056902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:19.056902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:19.056915 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:19.059053 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:19.059554 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:19.059558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:19.059558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:19.059663 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:19.059914 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:19.059915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:19.059915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:19.060222 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:19.060367 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:19.060532 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 04:58:19.060631 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":109,"threads_started":1}
I20260928 04:58:19.061933 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:19.071110 21707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:19.063425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:57593 (local address 127.0.0.1:38559)
0928 04:58:19.063478 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:19.063480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:19.063509 (+ 29us) server_negotiation.cc:407] Connection header received
0928 04:58:19.063576 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:19.063576 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:19.063587 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:19.063638 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:19.064331 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:19.065070 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:19.066139 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:19.066463 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:19.066579 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:19.068149 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:19.068151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:19.068151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:19.068159 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:19.069782 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:19.070238 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:19.070239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:19.070239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:19.070295 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:19.070542 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:19.070543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:19.070543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:19.070679 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:19.070776 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:19.070938 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 04:58:19.071011 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 04:58:19.079021 21715 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:19.079595 21715 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:19.082971 21715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52ec7ddf82584de4a8eca0466b327c42: Generated new TSK 0
I20260928 04:58:19.083211 21715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:19.086599 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 21635
W20260928 04:58:19.091852 21620 connection.cc:535] client connection to 127.0.0.1:38559 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38559): Connection reset by peer (error 104)
2026-09-28T04:58:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:19 dist-test-slave-m39l krb5kdc[21734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:19 dist-test-slave-m39l krb5kdc[21734](info): set up 2 sockets
Sep 28 04:58:19 dist-test-slave-m39l krb5kdc[21734](info): commencing operation
krb5kdc: starting...
W20260928 04:58:20.222822 392 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:20 dist-test-slave-m39l krb5kdc[21734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:58:20.353734 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:20.436766 21747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:20.436996 21747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:20.437021 21747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:20.438230 21747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:20.438277 21747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:20.438292 21747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:20.438304 21747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:20.438316 21747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:20.439937 21747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36273
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:20.440212 21747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:20.440465 21747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:20.445853 21755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:20.445981 21747 server_base.cc:1034] running on GCE node
W20260928 04:58:20.445842 21752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:20.445842 21753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:20.446365 21747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:20.446705 21747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:20.447877 21747 hybrid_clock.cc:648] HybridClock initialized: now 1790571500447879 us; error 78 us; skew 500 ppm
Sep 28 04:58:20 dist-test-slave-m39l krb5kdc[21734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:20.450704 21747 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:58:20.451263 21747 webserver.cc:460] Webserver started at http://127.0.0.1:41727/ using document root <none> and password file <none>
I20260928 04:58:20.451551 21747 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:20.451639 21747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:20.451805 21747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:20.452972 21747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "18627a98546c407f991a5aa19e20c956"
format_stamp: "Formatted at 2026-09-28 04:58:20 on dist-test-slave-m39l"
server_key: "dc4d00a7c3a0e24cf2c6d30c7148f64a"
server_key_iv: "b0eba1a8a3e59c602fcbcc8e2e5b566f"
server_key_version: "encryptionkey@0"
I20260928 04:58:20.453397 21747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "18627a98546c407f991a5aa19e20c956"
format_stamp: "Formatted at 2026-09-28 04:58:20 on dist-test-slave-m39l"
server_key: "dc4d00a7c3a0e24cf2c6d30c7148f64a"
server_key_iv: "b0eba1a8a3e59c602fcbcc8e2e5b566f"
server_key_version: "encryptionkey@0"
I20260928 04:58:20.454900 21747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260928 04:58:20.455765 21762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:20.456036 21747 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:20.456149 21747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "18627a98546c407f991a5aa19e20c956"
format_stamp: "Formatted at 2026-09-28 04:58:20 on dist-test-slave-m39l"
server_key: "dc4d00a7c3a0e24cf2c6d30c7148f64a"
server_key_iv: "b0eba1a8a3e59c602fcbcc8e2e5b566f"
server_key_version: "encryptionkey@0"
I20260928 04:58:20.456230 21747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:20.489425 21747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:20.493470 21747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:20.493770 21747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:20.499894 21747 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36273
I20260928 04:58:20.499961 21814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36273 every 8 connection(s)
I20260928 04:58:20.500509 21747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:20.501983 21815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:20.505575 21815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956: Bootstrap starting.
I20260928 04:58:20.506368 21815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:20.506927 21815 log.cc:824] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:20.507979 21815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956: No bootstrap required, opened a new log
I20260928 04:58:20.509238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 21747
I20260928 04:58:20.509356 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:20.509538 392 external_mini_cluster.cc:1428] Setting key f6672a8de98ac866d8ecf9265b62dc60
I20260928 04:58:20.510877 21815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18627a98546c407f991a5aa19e20c956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36273 } }
I20260928 04:58:20.511067 21815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:20.511108 21815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18627a98546c407f991a5aa19e20c956, State: Initialized, Role: FOLLOWER
I20260928 04:58:20.511243 21815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18627a98546c407f991a5aa19e20c956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36273 } }
I20260928 04:58:20.511339 21815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:20.511386 21815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:20.511440 21815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:20.512261 21815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18627a98546c407f991a5aa19e20c956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36273 } }
I20260928 04:58:20.512388 21815 leader_election.cc:302] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18627a98546c407f991a5aa19e20c956; no voters:
I20260928 04:58:20.512586 21815 leader_election.cc:288] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:20.512606 21820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:20.512802 21820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956 [term 1 LEADER]: Becoming Leader. State: Replica: 18627a98546c407f991a5aa19e20c956, State: Running, Role: LEADER
I20260928 04:58:20.512876 21815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:20.513129 21820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18627a98546c407f991a5aa19e20c956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36273 } }
I20260928 04:58:20.513597 21822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18627a98546c407f991a5aa19e20c956. Latest consensus state: current_term: 1 leader_uuid: "18627a98546c407f991a5aa19e20c956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18627a98546c407f991a5aa19e20c956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36273 } } }
I20260928 04:58:20.513701 21822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:20.514043 21821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18627a98546c407f991a5aa19e20c956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18627a98546c407f991a5aa19e20c956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36273 } } }
I20260928 04:58:20.514117 21821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956 [sys.catalog]: This master's current role is: LEADER
W20260928 04:58:20.515259 21835 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:58:20.515313 21835 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:58:20.515658 21829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:20.516439 21829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:20.516990 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:20.510345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:33863 (local address 127.0.0.1:36273)
0928 04:58:20.510579 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:20.510589 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:20.510628 (+ 39us) server_negotiation.cc:407] Connection header received
0928 04:58:20.510845 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:20.510854 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:20.511002 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:20.511543 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:20.511666 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:20.512941 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:20.514052 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:20.514774 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:20.515294 (+ 520us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:20.515872 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:20.515877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:20.515877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:20.516093 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:20.516134 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:20.516150 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:20.516687 (+ 537us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:20.516732 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 04:58:20.516762 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260928 04:58:20.517989 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:20.520147 21829 catalog_manager.cc:1347] Generated new cluster ID: 69c880990f344541b1caef1e7136d5e7
I20260928 04:58:20.520215 21829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:58:20.524585 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:20.519837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:56827 (local address 127.0.0.1:36273)
0928 04:58:20.519923 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:20.519925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:20.520260 (+ 335us) server_negotiation.cc:407] Connection header received
0928 04:58:20.520351 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:20.520353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:20.520381 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:20.520468 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:20.520603 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:20.521463 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:20.522519 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:20.523398 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:20.523495 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:20.524250 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:20.524251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:20.524251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:20.524258 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:20.524276 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:20.524282 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:20.524358 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:20.524484 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 04:58:20.524499 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 04:58:20.529392 21829 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:20.530148 21829 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:20.536864 21829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18627a98546c407f991a5aa19e20c956: Generated new TSK 0
I20260928 04:58:20.537132 21829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:20.540383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 21747
W20260928 04:58:20.545423 21732 connection.cc:535] client connection to 127.0.0.1:36273 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36273): Connection reset by peer (error 104)
2026-09-28T04:58:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:20 dist-test-slave-m39l krb5kdc[21846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:20 dist-test-slave-m39l krb5kdc[21846](info): set up 2 sockets
Sep 28 04:58:20 dist-test-slave-m39l krb5kdc[21846](info): commencing operation
krb5kdc: starting...
W20260928 04:58:21.631201 392 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:21 dist-test-slave-m39l krb5kdc[21846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:58:21.763015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:21.853487 21859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:21.853789 21859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:21.853837 21859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:21.855276 21859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:21.855371 21859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:21.855396 21859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:21.855417 21859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:21.855448 21859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:21.857177 21859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43547
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:21.857499 21859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:21.857750 21859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:21.863209 21867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:21.863209 21864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:21.863204 21865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:21.863525 21859 server_base.cc:1034] running on GCE node
I20260928 04:58:21.863771 21859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:21.864040 21859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:21.865188 21859 hybrid_clock.cc:648] HybridClock initialized: now 1790571501865173 us; error 28 us; skew 500 ppm
Sep 28 04:58:21 dist-test-slave-m39l krb5kdc[21846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:21.868000 21859 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:58:21.868566 21859 webserver.cc:460] Webserver started at http://127.0.0.1:34771/ using document root <none> and password file <none>
I20260928 04:58:21.868883 21859 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:21.869005 21859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:21.869167 21859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:21.870290 21859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "63ef0927006c4df7912fa1a7746c40a5"
format_stamp: "Formatted at 2026-09-28 04:58:21 on dist-test-slave-m39l"
server_key: "82afe99fdb4f0ea7981c55ed44aeda2e"
server_key_iv: "c592d484224865bee7db63401c68c890"
server_key_version: "encryptionkey@0"
I20260928 04:58:21.870726 21859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "63ef0927006c4df7912fa1a7746c40a5"
format_stamp: "Formatted at 2026-09-28 04:58:21 on dist-test-slave-m39l"
server_key: "82afe99fdb4f0ea7981c55ed44aeda2e"
server_key_iv: "c592d484224865bee7db63401c68c890"
server_key_version: "encryptionkey@0"
I20260928 04:58:21.872228 21859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:58:21.873099 21874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:21.873374 21859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:58:21.873484 21859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "63ef0927006c4df7912fa1a7746c40a5"
format_stamp: "Formatted at 2026-09-28 04:58:21 on dist-test-slave-m39l"
server_key: "82afe99fdb4f0ea7981c55ed44aeda2e"
server_key_iv: "c592d484224865bee7db63401c68c890"
server_key_version: "encryptionkey@0"
I20260928 04:58:21.873574 21859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:21.877257 21859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:21.879520 21859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:21.879695 21859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:21.884163 21859 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43547
I20260928 04:58:21.884197 21926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43547 every 8 connection(s)
I20260928 04:58:21.884575 21859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:21.885634 21927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:21.888156 21927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5: Bootstrap starting.
I20260928 04:58:21.888602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 21859
I20260928 04:58:21.888723 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:21.888916 392 external_mini_cluster.cc:1428] Setting key a885c3b5f165248db2367fc76e84f004
I20260928 04:58:21.888916 21927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:21.889217 21927 log.cc:824] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:21.890363 21927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5: No bootstrap required, opened a new log
I20260928 04:58:21.891758 21931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:21.889518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:44279 (local address 127.0.0.1:43547)
0928 04:58:21.889714 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:21.889724 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:21.889762 (+ 38us) server_negotiation.cc:407] Connection header received
0928 04:58:21.890313 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:21.890321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:21.890350 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:21.890559 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:21.891285 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:21.891289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:21.891289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:21.891293 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:21.891333 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:21.891344 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:21.891418 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:21.891498 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 04:58:21.891565 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260928 04:58:21.892999 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:21.893112 21927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ef0927006c4df7912fa1a7746c40a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43547 } }
I20260928 04:58:21.893277 21927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:21.893301 21927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ef0927006c4df7912fa1a7746c40a5, State: Initialized, Role: FOLLOWER
I20260928 04:58:21.893448 21927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ef0927006c4df7912fa1a7746c40a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43547 } }
I20260928 04:58:21.893533 21927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:21.893556 21927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:21.893623 21927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:21.894272 21927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ef0927006c4df7912fa1a7746c40a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43547 } }
I20260928 04:58:21.894388 21927 leader_election.cc:302] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63ef0927006c4df7912fa1a7746c40a5; no voters:
I20260928 04:58:21.894585 21927 leader_election.cc:288] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:21.894622 21936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:21.894783 21936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5 [term 1 LEADER]: Becoming Leader. State: Replica: 63ef0927006c4df7912fa1a7746c40a5, State: Running, Role: LEADER
I20260928 04:58:21.894850 21927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:21.894881 21936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ef0927006c4df7912fa1a7746c40a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43547 } }
I20260928 04:58:21.895289 21937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63ef0927006c4df7912fa1a7746c40a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ef0927006c4df7912fa1a7746c40a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43547 } } }
I20260928 04:58:21.895406 21937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:21.896013 21945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:21.896546 21945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:21.896875 21931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:21.894618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:49751 (local address 127.0.0.1:43547)
0928 04:58:21.894661 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:21.894663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:21.895701 (+ 1038us) server_negotiation.cc:407] Connection header received
0928 04:58:21.895769 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:21.895770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:21.895777 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:21.895827 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:21.896601 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:21.896602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:21.896602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:21.896607 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:21.896623 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:21.896627 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:21.896659 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:21.896777 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 04:58:21.896794 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:58:21.895987 21938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63ef0927006c4df7912fa1a7746c40a5. Latest consensus state: current_term: 1 leader_uuid: "63ef0927006c4df7912fa1a7746c40a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ef0927006c4df7912fa1a7746c40a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43547 } } }
I20260928 04:58:21.897308 21938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:21.898350 21945 catalog_manager.cc:1347] Generated new cluster ID: dfe83084248947f9bd3ffaab0d91a2ae
I20260928 04:58:21.898403 21945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:58:21.923925 21945 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:21.924135 21945 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:21.930728 21945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63ef0927006c4df7912fa1a7746c40a5: Generated new TSK 0
I20260928 04:58:21.930960 21945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:21.940272 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 21859
2026-09-28T04:58:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:21 dist-test-slave-m39l krb5kdc[21958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:21 dist-test-slave-m39l krb5kdc[21958](info): set up 2 sockets
Sep 28 04:58:21 dist-test-slave-m39l krb5kdc[21958](info): commencing operation
krb5kdc: starting...
W20260928 04:58:23.018810 392 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:23 dist-test-slave-m39l krb5kdc[21958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:58:23.150480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36353
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:23.236158 21971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:23.236374 21971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:23.236400 21971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:23.237572 21971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:23.237609 21971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:23.237624 21971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:23.237637 21971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:23.237649 21971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:23.239153 21971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36353
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:23.239356 21971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:23.239548 21971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:23.244874 21977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:23.244845 21979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:23.245007 21971 server_base.cc:1034] running on GCE node
W20260928 04:58:23.244831 21976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:23.245335 21971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:23.245636 21971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:23.246795 21971 hybrid_clock.cc:648] HybridClock initialized: now 1790571503246804 us; error 80 us; skew 500 ppm
Sep 28 04:58:23 dist-test-slave-m39l krb5kdc[21958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:23.256850 21971 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:58:23.257412 21971 webserver.cc:460] Webserver started at http://127.0.98.62:45619/ using document root <none> and password file <none>
I20260928 04:58:23.257701 21971 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:23.257757 21971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:23.257921 21971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:23.259325 21971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "75626f6eb6ba4e2fa00b32ce6e31e60d"
format_stamp: "Formatted at 2026-09-28 04:58:23 on dist-test-slave-m39l"
server_key: "2af8714beff202f8f1f32d1c166f3f53"
server_key_iv: "ebf1476695bcc3ce2aafe83769642eea"
server_key_version: "encryptionkey@0"
I20260928 04:58:23.259773 21971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "75626f6eb6ba4e2fa00b32ce6e31e60d"
format_stamp: "Formatted at 2026-09-28 04:58:23 on dist-test-slave-m39l"
server_key: "2af8714beff202f8f1f32d1c166f3f53"
server_key_iv: "ebf1476695bcc3ce2aafe83769642eea"
server_key_version: "encryptionkey@0"
I20260928 04:58:23.261739 21971 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 04:58:23.262722 21986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:23.262921 21971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:23.263028 21971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "75626f6eb6ba4e2fa00b32ce6e31e60d"
format_stamp: "Formatted at 2026-09-28 04:58:23 on dist-test-slave-m39l"
server_key: "2af8714beff202f8f1f32d1c166f3f53"
server_key_iv: "ebf1476695bcc3ce2aafe83769642eea"
server_key_version: "encryptionkey@0"
I20260928 04:58:23.263094 21971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:23.273226 21971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:23.275692 21971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:23.275823 21971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:23.279970 21971 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36353
I20260928 04:58:23.280004 22038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36353 every 8 connection(s)
I20260928 04:58:23.280385 21971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:23.281384 22039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:23.283996 22039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d: Bootstrap starting.
I20260928 04:58:23.284621 22039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:23.284941 22039 log.cc:824] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:23.285774 22039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d: No bootstrap required, opened a new log
I20260928 04:58:23.285993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 21971
I20260928 04:58:23.286118 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:23.286291 392 external_mini_cluster.cc:1428] Setting key 00d25b61c5d828d2dbd907363c451579
I20260928 04:58:23.288383 22039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75626f6eb6ba4e2fa00b32ce6e31e60d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36353 } }
I20260928 04:58:23.288618 22039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:23.288655 22039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75626f6eb6ba4e2fa00b32ce6e31e60d, State: Initialized, Role: FOLLOWER
I20260928 04:58:23.288796 22039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75626f6eb6ba4e2fa00b32ce6e31e60d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36353 } }
I20260928 04:58:23.288923 22039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:23.288959 22039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:23.289000 22039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:23.289667 22039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75626f6eb6ba4e2fa00b32ce6e31e60d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36353 } }
I20260928 04:58:23.289855 22039 leader_election.cc:302] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75626f6eb6ba4e2fa00b32ce6e31e60d; no voters:
I20260928 04:58:23.290030 22039 leader_election.cc:288] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:23.290099 22044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:23.290253 22044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d [term 1 LEADER]: Becoming Leader. State: Replica: 75626f6eb6ba4e2fa00b32ce6e31e60d, State: Running, Role: LEADER
I20260928 04:58:23.290318 22039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:23.290341 22044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75626f6eb6ba4e2fa00b32ce6e31e60d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36353 } }
I20260928 04:58:23.290723 22044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75626f6eb6ba4e2fa00b32ce6e31e60d. Latest consensus state: current_term: 1 leader_uuid: "75626f6eb6ba4e2fa00b32ce6e31e60d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75626f6eb6ba4e2fa00b32ce6e31e60d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36353 } } }
I20260928 04:58:23.290817 22044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:23.290733 22045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75626f6eb6ba4e2fa00b32ce6e31e60d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75626f6eb6ba4e2fa00b32ce6e31e60d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36353 } } }
I20260928 04:58:23.290953 22045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:23.291556 22048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:23.292037 22048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:23.294082 22048 catalog_manager.cc:1347] Generated new cluster ID: 0642686f4ade414cbf0756d7222b0a83
I20260928 04:58:23.294143 22048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:58:23 dist-test-slave-m39l krb5kdc[21958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:58:23.301431 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:23.286910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39473 (local address 127.0.98.62:36353)
0928 04:58:23.287123 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:23.287133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:23.287169 (+ 36us) server_negotiation.cc:407] Connection header received
0928 04:58:23.288614 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:23.288623 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:23.288697 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:23.288862 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:23.289596 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:23.291322 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:23.292434 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:23.293235 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:23.293483 (+ 248us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:23.296961 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:23.296966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:23.296967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:23.296981 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:23.299636 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:23.300138 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:23.300141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:23.300141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:23.300252 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:23.300503 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:23.300504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:23.300504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:23.300813 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:23.300971 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:23.301113 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 04:58:23.301210 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":98,"threads_started":1}
I20260928 04:58:23.302814 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:23.315912 22048 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:23.316537 22048 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:23.323724 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:23.304306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44799 (local address 127.0.98.62:36353)
0928 04:58:23.304343 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:23.304345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:23.315732 (+ 11387us) server_negotiation.cc:407] Connection header received
0928 04:58:23.315861 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:23.315863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:23.315878 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:23.315955 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:23.316778 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:23.317527 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:23.318633 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:23.318939 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:23.319013 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:23.320636 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:23.320638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:23.320638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:23.320646 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:23.322317 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:23.322845 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:23.322846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:23.322847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:23.322908 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:23.323158 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:23.323159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:23.323160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:23.323287 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:23.323388 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:23.323556 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 04:58:23.323621 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 04:58:23.326313 22048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75626f6eb6ba4e2fa00b32ce6e31e60d: Generated new TSK 0
I20260928 04:58:23.326565 22048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:23.329623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 21971
W20260928 04:58:23.333841 21956 connection.cc:535] client connection to 127.0.98.62:36353 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36353): Connection reset by peer (error 104)
2026-09-28T04:58:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:23 dist-test-slave-m39l krb5kdc[22070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:23 dist-test-slave-m39l krb5kdc[22070](info): set up 2 sockets
Sep 28 04:58:23 dist-test-slave-m39l krb5kdc[22070](info): commencing operation
krb5kdc: starting...
W20260928 04:58:24.394872 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:24 dist-test-slave-m39l krb5kdc[22070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:58:24.521522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40127
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:24.605929 22083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:24.606206 22083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:24.606246 22083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:24.607523 22083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:24.607605 22083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:24.607628 22083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:24.607647 22083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:24.607679 22083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:24.609701 22083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40127
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:24.610040 22083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:24.610293 22083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:24.615466 22088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:24.615466 22091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:24.615511 22089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:24.615602 22083 server_base.cc:1034] running on GCE node
I20260928 04:58:24.616043 22083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:24.616394 22083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:24.617578 22083 hybrid_clock.cc:648] HybridClock initialized: now 1790571504617552 us; error 42 us; skew 500 ppm
Sep 28 04:58:24 dist-test-slave-m39l krb5kdc[22070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:24.620484 22083 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:58:24.621027 22083 webserver.cc:460] Webserver started at http://127.0.98.62:35267/ using document root <none> and password file <none>
I20260928 04:58:24.621264 22083 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:24.621335 22083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:24.621516 22083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:24.622731 22083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "3f6827dbc35a46e8b7dd32f26a3c4c2e"
format_stamp: "Formatted at 2026-09-28 04:58:24 on dist-test-slave-m39l"
server_key: "aaedfaffd94c593b8da7884ccecb6177"
server_key_iv: "7827c2aff10f6034a728fdd3e6d6fc05"
server_key_version: "encryptionkey@0"
I20260928 04:58:24.623107 22083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "3f6827dbc35a46e8b7dd32f26a3c4c2e"
format_stamp: "Formatted at 2026-09-28 04:58:24 on dist-test-slave-m39l"
server_key: "aaedfaffd94c593b8da7884ccecb6177"
server_key_iv: "7827c2aff10f6034a728fdd3e6d6fc05"
server_key_version: "encryptionkey@0"
I20260928 04:58:24.624558 22083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:58:24.625487 22098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:24.625741 22083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:24.625859 22083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "3f6827dbc35a46e8b7dd32f26a3c4c2e"
format_stamp: "Formatted at 2026-09-28 04:58:24 on dist-test-slave-m39l"
server_key: "aaedfaffd94c593b8da7884ccecb6177"
server_key_iv: "7827c2aff10f6034a728fdd3e6d6fc05"
server_key_version: "encryptionkey@0"
I20260928 04:58:24.625948 22083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:24.636198 22083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:24.638479 22083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:24.638689 22083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:24.643136 22083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40127
I20260928 04:58:24.643198 22150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40127 every 8 connection(s)
I20260928 04:58:24.643586 22083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:24.644532 22151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:24.646781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 22083
I20260928 04:58:24.646950 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:24.647168 392 external_mini_cluster.cc:1428] Setting key 80c7d0d5f3667311a78da266e4e14b5d
I20260928 04:58:24.647308 22151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e: Bootstrap starting.
I20260928 04:58:24.648170 22151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:24.648535 22151 log.cc:824] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:24.649291 22151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e: No bootstrap required, opened a new log
I20260928 04:58:24.651290 22151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6827dbc35a46e8b7dd32f26a3c4c2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } }
I20260928 04:58:24.651486 22151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:24.651535 22151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f6827dbc35a46e8b7dd32f26a3c4c2e, State: Initialized, Role: FOLLOWER
I20260928 04:58:24.651677 22151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6827dbc35a46e8b7dd32f26a3c4c2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } }
I20260928 04:58:24.651779 22151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:24.651825 22151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:24.651875 22151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:24.652683 22151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6827dbc35a46e8b7dd32f26a3c4c2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } }
I20260928 04:58:24.652823 22151 leader_election.cc:302] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f6827dbc35a46e8b7dd32f26a3c4c2e; no voters:
I20260928 04:58:24.653018 22151 leader_election.cc:288] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:24.653081 22156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:24.653246 22156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e [term 1 LEADER]: Becoming Leader. State: Replica: 3f6827dbc35a46e8b7dd32f26a3c4c2e, State: Running, Role: LEADER
I20260928 04:58:24.653303 22151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:24.653272 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:24.647722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58417 (local address 127.0.98.62:40127)
0928 04:58:24.647950 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:24.647960 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:24.647997 (+ 37us) server_negotiation.cc:407] Connection header received
0928 04:58:24.648638 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:24.648647 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:24.648720 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:24.648921 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:24.649112 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:24.650101 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:24.651035 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:24.651485 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:24.651989 (+ 504us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:24.652428 (+ 439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:24.652432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:24.652432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:24.652614 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:24.652648 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:24.652661 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:24.652736 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:24.652966 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 04:58:24.652997 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":101,"threads_started":1}
I20260928 04:58:24.653386 22156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6827dbc35a46e8b7dd32f26a3c4c2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } }
I20260928 04:58:24.653973 22158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f6827dbc35a46e8b7dd32f26a3c4c2e. Latest consensus state: current_term: 1 leader_uuid: "3f6827dbc35a46e8b7dd32f26a3c4c2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6827dbc35a46e8b7dd32f26a3c4c2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } }
I20260928 04:58:24.653990 22157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f6827dbc35a46e8b7dd32f26a3c4c2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6827dbc35a46e8b7dd32f26a3c4c2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40127 } } }
I20260928 04:58:24.654086 22158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:24.654106 22157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:24.654297 22165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:24.654760 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:24.654923 22165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:24.659821 22165 catalog_manager.cc:1347] Generated new cluster ID: 181a54fb67de4868a3ab8a20bde34698
I20260928 04:58:24.659909 22165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:58:24.666666 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:24.656407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42447 (local address 127.0.98.62:40127)
0928 04:58:24.656452 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:24.656455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:24.656575 (+ 120us) server_negotiation.cc:407] Connection header received
0928 04:58:24.656637 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:24.656638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:24.656646 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:24.656693 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:24.656823 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:24.657301 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:24.658348 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:24.658657 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:24.658740 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:24.659581 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:24.659582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:24.659583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:24.659588 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:24.659602 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:24.659606 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:24.659680 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:24.659746 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 04:58:24.659757 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:58:24.701237 22165 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:24.701877 22165 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:24.704941 22165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f6827dbc35a46e8b7dd32f26a3c4c2e: Generated new TSK 0
I20260928 04:58:24.705157 22165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:24.713454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 22083
W20260928 04:58:24.718348 22068 connection.cc:535] client connection to 127.0.98.62:40127 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40127): Connection reset by peer (error 104)
2026-09-28T04:58:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:24 dist-test-slave-m39l krb5kdc[22182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:24 dist-test-slave-m39l krb5kdc[22182](info): set up 2 sockets
Sep 28 04:58:24 dist-test-slave-m39l krb5kdc[22182](info): commencing operation
krb5kdc: starting...
W20260928 04:58:25.787209 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:25 dist-test-slave-m39l krb5kdc[22182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:58:25.915359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34095
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:25.998872 22195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:25.999104 22195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:25.999128 22195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:26.000296 22195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:26.000339 22195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:26.000352 22195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:26.000365 22195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:26.000378 22195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:26.001849 22195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34095
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:26.002066 22195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:26.002252 22195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:26.007393 22203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:26.007397 22201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:26.007393 22200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:26.007699 22195 server_base.cc:1034] running on GCE node
I20260928 04:58:26.008021 22195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:26.008364 22195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:26.009577 22195 hybrid_clock.cc:648] HybridClock initialized: now 1790571506009545 us; error 50 us; skew 500 ppm
Sep 28 04:58:26 dist-test-slave-m39l krb5kdc[22182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:26.012725 22195 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:58:26.013350 22195 webserver.cc:460] Webserver started at http://127.0.98.62:35649/ using document root <none> and password file <none>
I20260928 04:58:26.013607 22195 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:26.013684 22195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:26.013852 22195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:26.015138 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "33fac06dd91d47f6b5c5a4c1ea80d7ff"
format_stamp: "Formatted at 2026-09-28 04:58:26 on dist-test-slave-m39l"
server_key: "2e8e28429dc6278daa9b2474469b1d53"
server_key_iv: "fe1c45f4777e9afbbf7e9b7ead7cf4bd"
server_key_version: "encryptionkey@0"
I20260928 04:58:26.015542 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "33fac06dd91d47f6b5c5a4c1ea80d7ff"
format_stamp: "Formatted at 2026-09-28 04:58:26 on dist-test-slave-m39l"
server_key: "2e8e28429dc6278daa9b2474469b1d53"
server_key_iv: "fe1c45f4777e9afbbf7e9b7ead7cf4bd"
server_key_version: "encryptionkey@0"
I20260928 04:58:26.017043 22195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:58:26.017936 22210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:26.018178 22195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:26.018306 22195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "33fac06dd91d47f6b5c5a4c1ea80d7ff"
format_stamp: "Formatted at 2026-09-28 04:58:26 on dist-test-slave-m39l"
server_key: "2e8e28429dc6278daa9b2474469b1d53"
server_key_iv: "fe1c45f4777e9afbbf7e9b7ead7cf4bd"
server_key_version: "encryptionkey@0"
I20260928 04:58:26.018389 22195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:26.022055 22195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:26.024427 22195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:26.024633 22195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:26.030412 22195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34095
I20260928 04:58:26.030471 22262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34095 every 8 connection(s)
I20260928 04:58:26.031085 22195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:26.032445 22263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:26.036465 22263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff: Bootstrap starting.
I20260928 04:58:26.037176 22263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:26.037518 22263 log.cc:824] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:26.038427 22263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff: No bootstrap required, opened a new log
I20260928 04:58:26.040261 22263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fac06dd91d47f6b5c5a4c1ea80d7ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34095 } }
I20260928 04:58:26.040464 22263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:26.040547 22263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33fac06dd91d47f6b5c5a4c1ea80d7ff, State: Initialized, Role: FOLLOWER
I20260928 04:58:26.040755 22263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fac06dd91d47f6b5c5a4c1ea80d7ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34095 } }
I20260928 04:58:26.040869 22263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:26.040920 22263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:26.040977 22263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:26.041020 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 22195
I20260928 04:58:26.041158 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:26.041375 392 external_mini_cluster.cc:1428] Setting key 04a40268b7ec0da780b10e5e6cb13779
I20260928 04:58:26.041836 22263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fac06dd91d47f6b5c5a4c1ea80d7ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34095 } }
I20260928 04:58:26.041994 22263 leader_election.cc:302] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33fac06dd91d47f6b5c5a4c1ea80d7ff; no voters:
I20260928 04:58:26.042191 22263 leader_election.cc:288] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:26.042254 22268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:26.042433 22268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff [term 1 LEADER]: Becoming Leader. State: Replica: 33fac06dd91d47f6b5c5a4c1ea80d7ff, State: Running, Role: LEADER
I20260928 04:58:26.042546 22263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:26.042601 22268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fac06dd91d47f6b5c5a4c1ea80d7ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34095 } }
I20260928 04:58:26.043087 22269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33fac06dd91d47f6b5c5a4c1ea80d7ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fac06dd91d47f6b5c5a4c1ea80d7ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34095 } } }
I20260928 04:58:26.043180 22269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:26.043438 22270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33fac06dd91d47f6b5c5a4c1ea80d7ff. Latest consensus state: current_term: 1 leader_uuid: "33fac06dd91d47f6b5c5a4c1ea80d7ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fac06dd91d47f6b5c5a4c1ea80d7ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34095 } } }
I20260928 04:58:26.043586 22270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:26.044317 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:26.042019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49773 (local address 127.0.98.62:34095)
0928 04:58:26.042254 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:26.042266 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:26.042307 (+ 41us) server_negotiation.cc:407] Connection header received
0928 04:58:26.042447 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:26.042455 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:26.042514 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:26.043011 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:26.043555 (+ 544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:26.043559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:26.043559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:26.043564 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:26.043616 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:26.043636 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:26.043959 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:26.043996 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 04:58:26.044048 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":112,"threads_started":1}
W20260928 04:58:26.044946 22284 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:58:26.045135 22284 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:58:26.045336 22272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:26.045537 22272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:26.045876 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:26.048884 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:26.047631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52123 (local address 127.0.98.62:34095)
0928 04:58:26.047679 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:26.047681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:26.047787 (+ 106us) server_negotiation.cc:407] Connection header received
0928 04:58:26.047857 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:26.047858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:26.047864 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:26.047927 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:26.048632 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:26.048633 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:26.048633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:26.048638 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:26.048655 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:26.048660 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:26.048711 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:26.048780 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 04:58:26.048806 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:58:26.051117 22272 catalog_manager.cc:1347] Generated new cluster ID: 8d52d0aed25c4702bc486d42813fa060
I20260928 04:58:26.051208 22272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:58:26.069329 22272 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:26.069561 22272 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:26.077131 22272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33fac06dd91d47f6b5c5a4c1ea80d7ff: Generated new TSK 0
I20260928 04:58:26.077409 22272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:26.079450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 22195
2026-09-28T04:58:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:26 dist-test-slave-m39l krb5kdc[22294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:26 dist-test-slave-m39l krb5kdc[22294](info): set up 2 sockets
Sep 28 04:58:26 dist-test-slave-m39l krb5kdc[22294](info): commencing operation
krb5kdc: starting...
W20260928 04:58:27.170678 392 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:27 dist-test-slave-m39l krb5kdc[22294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:58:27.304034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37265
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:27.388975 22307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:27.389249 22307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:27.389285 22307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:27.390533 22307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:27.390586 22307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:27.390601 22307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:27.390614 22307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:27.390627 22307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:27.392163 22307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:27.392387 22307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:27.392584 22307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:27.398366 22315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:27.398352 22312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:27.398376 22313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:27.398543 22307 server_base.cc:1034] running on GCE node
I20260928 04:58:27.399062 22307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:27.399340 22307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:27.400539 22307 hybrid_clock.cc:648] HybridClock initialized: now 1790571507400527 us; error 30 us; skew 500 ppm
Sep 28 04:58:27 dist-test-slave-m39l krb5kdc[22294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:27.403455 22307 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:58:27.404034 22307 webserver.cc:460] Webserver started at http://127.0.98.62:35933/ using document root <none> and password file <none>
I20260928 04:58:27.404289 22307 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:27.404361 22307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:27.404544 22307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:27.405704 22307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "18385c87b7d14410bb36e5b7c85b4f4f"
format_stamp: "Formatted at 2026-09-28 04:58:27 on dist-test-slave-m39l"
server_key: "3e5eb1e829408f6684863cd849556e26"
server_key_iv: "b83d9c357e05523940cb6a26c088f0fa"
server_key_version: "encryptionkey@0"
I20260928 04:58:27.406119 22307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "18385c87b7d14410bb36e5b7c85b4f4f"
format_stamp: "Formatted at 2026-09-28 04:58:27 on dist-test-slave-m39l"
server_key: "3e5eb1e829408f6684863cd849556e26"
server_key_iv: "b83d9c357e05523940cb6a26c088f0fa"
server_key_version: "encryptionkey@0"
I20260928 04:58:27.407701 22307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:58:27.408658 22322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:27.408987 22307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:27.409117 22307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "18385c87b7d14410bb36e5b7c85b4f4f"
format_stamp: "Formatted at 2026-09-28 04:58:27 on dist-test-slave-m39l"
server_key: "3e5eb1e829408f6684863cd849556e26"
server_key_iv: "b83d9c357e05523940cb6a26c088f0fa"
server_key_version: "encryptionkey@0"
I20260928 04:58:27.409183 22307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:27.428440 22307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:27.431103 22307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:27.431232 22307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:27.435487 22307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37265
I20260928 04:58:27.435523 22374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37265 every 8 connection(s)
I20260928 04:58:27.435940 22307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:27.436992 22375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:27.439670 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 22307
I20260928 04:58:27.439777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:27.439760 22375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f: Bootstrap starting.
I20260928 04:58:27.439980 392 external_mini_cluster.cc:1428] Setting key 14749bc2036aa54caeac16f2637f440c
I20260928 04:58:27.440691 22375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:27.441107 22375 log.cc:824] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:27.442108 22375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f: No bootstrap required, opened a new log
I20260928 04:58:27.444689 22375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18385c87b7d14410bb36e5b7c85b4f4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37265 } }
I20260928 04:58:27.444901 22375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:27.444937 22375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18385c87b7d14410bb36e5b7c85b4f4f, State: Initialized, Role: FOLLOWER
I20260928 04:58:27.445081 22375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18385c87b7d14410bb36e5b7c85b4f4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37265 } }
I20260928 04:58:27.445182 22375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:27.445257 22375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:27.445314 22375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:27.446099 22375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18385c87b7d14410bb36e5b7c85b4f4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37265 } }
I20260928 04:58:27.446220 22375 leader_election.cc:302] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18385c87b7d14410bb36e5b7c85b4f4f; no voters:
I20260928 04:58:27.446476 22375 leader_election.cc:288] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:27.446532 22380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:27.446733 22380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f [term 1 LEADER]: Becoming Leader. State: Replica: 18385c87b7d14410bb36e5b7c85b4f4f, State: Running, Role: LEADER
I20260928 04:58:27.446810 22375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:27.446916 22380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18385c87b7d14410bb36e5b7c85b4f4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37265 } }
I20260928 04:58:27.447340 22382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18385c87b7d14410bb36e5b7c85b4f4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18385c87b7d14410bb36e5b7c85b4f4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37265 } } }
I20260928 04:58:27.447332 22380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18385c87b7d14410bb36e5b7c85b4f4f. Latest consensus state: current_term: 1 leader_uuid: "18385c87b7d14410bb36e5b7c85b4f4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18385c87b7d14410bb36e5b7c85b4f4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37265 } } }
I20260928 04:58:27.447440 22382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:27.447448 22380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:27.448367 22388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 04:58:27 dist-test-slave-m39l krb5kdc[22294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:58:27.448954 22388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:27.450796 22388 catalog_manager.cc:1347] Generated new cluster ID: c755a8968c5a465cb4724bba38bcb549
I20260928 04:58:27.450857 22388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:58:27.453128 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:27.440754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35981 (local address 127.0.98.62:37265)
0928 04:58:27.441036 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:27.441044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:27.441072 (+ 28us) server_negotiation.cc:407] Connection header received
0928 04:58:27.441748 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:27.441758 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:27.441817 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:27.441949 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:27.442948 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:27.443914 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:27.445162 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:27.445739 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:27.445825 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:27.449144 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:27.449148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:27.449149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:27.449162 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:27.451435 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:27.451855 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:27.451859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:27.451859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:27.451965 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:27.452181 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:27.452182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:27.452182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:27.452453 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:27.452629 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:27.452742 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 04:58:27.452893 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":181,"threads_started":1}
I20260928 04:58:27.454033 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:27.462085 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:27.455650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60031 (local address 127.0.98.62:37265)
0928 04:58:27.455688 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:27.455690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:27.455698 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:58:27.455719 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:27.455719 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:27.455729 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:27.455779 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:27.456321 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:27.456824 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:27.457565 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:27.457830 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:27.457924 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:27.459336 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:27.459337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:27.459338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:27.459346 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:27.460886 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:27.461266 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:27.461267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:27.461267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:27.461332 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:27.461550 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:27.461551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:27.461552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:27.461674 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:27.461788 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:27.461890 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 04:58:27.461970 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:58:27.471262 22388 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:27.471896 22388 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:27.483569 22388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18385c87b7d14410bb36e5b7c85b4f4f: Generated new TSK 0
I20260928 04:58:27.483832 22388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:27.488019 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 22307
2026-09-28T04:58:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:27 dist-test-slave-m39l krb5kdc[22406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:27 dist-test-slave-m39l krb5kdc[22406](info): set up 2 sockets
Sep 28 04:58:27 dist-test-slave-m39l krb5kdc[22406](info): commencing operation
krb5kdc: starting...
W20260928 04:58:28.538863 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:28 dist-test-slave-m39l krb5kdc[22406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:58:28.671603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45735
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:28.756481 22419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:28.756779 22419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:28.756825 22419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:28.758123 22419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:28.758220 22419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:28.758246 22419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:28.758270 22419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:28.758299 22419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:28.759994 22419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:28.760334 22419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:28.760599 22419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:28.766292 22425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:28.766429 22419 server_base.cc:1034] running on GCE node
W20260928 04:58:28.766287 22424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:28.766287 22427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:28.766777 22419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:28.767055 22419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:28.768229 22419 hybrid_clock.cc:648] HybridClock initialized: now 1790571508768214 us; error 30 us; skew 500 ppm
Sep 28 04:58:28 dist-test-slave-m39l krb5kdc[22406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:28.780036 22419 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:58:28.780596 22419 webserver.cc:460] Webserver started at http://127.0.98.62:43773/ using document root <none> and password file <none>
I20260928 04:58:28.780844 22419 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:28.780925 22419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:28.781080 22419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:28.782222 22419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "92010140de4d431693008622fcbc3a0d"
format_stamp: "Formatted at 2026-09-28 04:58:28 on dist-test-slave-m39l"
server_key: "32bb723486af79ee21b865161d68516e"
server_key_iv: "f4ef658db25aa4e49f21a35db00d4c11"
server_key_version: "encryptionkey@0"
I20260928 04:58:28.782672 22419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "92010140de4d431693008622fcbc3a0d"
format_stamp: "Formatted at 2026-09-28 04:58:28 on dist-test-slave-m39l"
server_key: "32bb723486af79ee21b865161d68516e"
server_key_iv: "f4ef658db25aa4e49f21a35db00d4c11"
server_key_version: "encryptionkey@0"
I20260928 04:58:28.784260 22419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:58:28.785235 22434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:28.785480 22419 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:28.785609 22419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "92010140de4d431693008622fcbc3a0d"
format_stamp: "Formatted at 2026-09-28 04:58:28 on dist-test-slave-m39l"
server_key: "32bb723486af79ee21b865161d68516e"
server_key_iv: "f4ef658db25aa4e49f21a35db00d4c11"
server_key_version: "encryptionkey@0"
I20260928 04:58:28.785704 22419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:28.797772 22419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:28.800518 22419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:28.800740 22419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:28.805656 22419 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45735
I20260928 04:58:28.805717 22486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45735 every 8 connection(s)
I20260928 04:58:28.806155 22419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:28.807152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 22419
I20260928 04:58:28.807283 22487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:28.807291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:28.807512 392 external_mini_cluster.cc:1428] Setting key 1891581eac8553c40b924f3c37427b44
I20260928 04:58:28.810839 22487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d: Bootstrap starting.
I20260928 04:58:28.811673 22487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:28.811985 22487 log.cc:824] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:28.812917 22487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d: No bootstrap required, opened a new log
I20260928 04:58:28.813838 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:28.808184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33617 (local address 127.0.98.62:45735)
0928 04:58:28.808400 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:28.808408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:28.808439 (+ 31us) server_negotiation.cc:407] Connection header received
0928 04:58:28.808925 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:28.808933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:28.808987 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:28.809118 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:28.809362 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:28.810278 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:28.811540 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:28.812097 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:28.812195 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:28.812984 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:28.812989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:28.812989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:28.813228 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:28.813284 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:28.813301 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:28.813388 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:28.813515 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 04:58:28.813598 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":117,"threads_started":1}
I20260928 04:58:28.815282 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:28.815639 22487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92010140de4d431693008622fcbc3a0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45735 } }
I20260928 04:58:28.815867 22487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:28.815910 22487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92010140de4d431693008622fcbc3a0d, State: Initialized, Role: FOLLOWER
I20260928 04:58:28.816047 22487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92010140de4d431693008622fcbc3a0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45735 } }
I20260928 04:58:28.816148 22487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:28.816182 22487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:28.816237 22487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:28.817137 22487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92010140de4d431693008622fcbc3a0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45735 } }
I20260928 04:58:28.817299 22487 leader_election.cc:302] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92010140de4d431693008622fcbc3a0d; no voters:
I20260928 04:58:28.817540 22487 leader_election.cc:288] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:28.817603 22497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:28.817857 22497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d [term 1 LEADER]: Becoming Leader. State: Replica: 92010140de4d431693008622fcbc3a0d, State: Running, Role: LEADER
I20260928 04:58:28.817955 22487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:28.818028 22497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92010140de4d431693008622fcbc3a0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45735 } }
I20260928 04:58:28.818492 22499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92010140de4d431693008622fcbc3a0d. Latest consensus state: current_term: 1 leader_uuid: "92010140de4d431693008622fcbc3a0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92010140de4d431693008622fcbc3a0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45735 } } }
I20260928 04:58:28.818639 22499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:28.818554 22498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92010140de4d431693008622fcbc3a0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92010140de4d431693008622fcbc3a0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45735 } } }
I20260928 04:58:28.818898 22498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:28.819229 22505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:28.819762 22505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:28.821112 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:28.817112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47175 (local address 127.0.98.62:45735)
0928 04:58:28.817162 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:28.817165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:28.817210 (+ 45us) server_negotiation.cc:407] Connection header received
0928 04:58:28.817341 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:28.817342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:28.817355 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:28.817441 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:28.817673 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:28.818573 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:28.819691 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:28.819992 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:28.820072 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:28.820843 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:28.820845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:28.820845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:28.820851 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:28.820878 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:28.820883 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:28.820942 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:28.821001 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 04:58:28.821029 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:58:28.823055 22505 catalog_manager.cc:1347] Generated new cluster ID: fc17a87c61184597bc5e8e2c0961fc90
I20260928 04:58:28.823119 22505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:58:28.841322 22505 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:28.841933 22505 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:28.844373 22505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92010140de4d431693008622fcbc3a0d: Generated new TSK 0
I20260928 04:58:28.844604 22505 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:28.853222 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 22419
2026-09-28T04:58:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:28 dist-test-slave-m39l krb5kdc[22518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:28 dist-test-slave-m39l krb5kdc[22518](info): set up 2 sockets
Sep 28 04:58:28 dist-test-slave-m39l krb5kdc[22518](info): commencing operation
krb5kdc: starting...
W20260928 04:58:29.914825 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:30 dist-test-slave-m39l krb5kdc[22518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:58:30.053819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45505
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:30.137826 22531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:30.138175 22531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:30.138231 22531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:30.140218 22531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:30.140313 22531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:30.140342 22531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:30.140363 22531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:30.140384 22531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:30.142336 22531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:30.142737 22531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:30.143011 22531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:30.148599 22537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:30.148648 22531 server_base.cc:1034] running on GCE node
W20260928 04:58:30.148562 22536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:30.148553 22539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:30.149013 22531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:30.149353 22531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:30.150589 22531 hybrid_clock.cc:648] HybridClock initialized: now 1790571510150572 us; error 36 us; skew 500 ppm
Sep 28 04:58:30 dist-test-slave-m39l krb5kdc[22518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:30.153434 22531 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:58:30.153927 22531 webserver.cc:460] Webserver started at http://127.0.98.62:35075/ using document root <none> and password file <none>
I20260928 04:58:30.154173 22531 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:30.154244 22531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:30.154422 22531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:30.155550 22531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "8d3e88e37d1f4912b6dceaa64dcfdec6"
format_stamp: "Formatted at 2026-09-28 04:58:30 on dist-test-slave-m39l"
server_key: "e157d537d3be7ac11c6d1453b0222857"
server_key_iv: "f527b88608e76c40ac1697b7276aab1c"
server_key_version: "encryptionkey@0"
I20260928 04:58:30.155915 22531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d3e88e37d1f4912b6dceaa64dcfdec6"
format_stamp: "Formatted at 2026-09-28 04:58:30 on dist-test-slave-m39l"
server_key: "e157d537d3be7ac11c6d1453b0222857"
server_key_iv: "f527b88608e76c40ac1697b7276aab1c"
server_key_version: "encryptionkey@0"
I20260928 04:58:30.157321 22531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:58:30.158255 22546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:30.158598 22531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:30.158722 22531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "8d3e88e37d1f4912b6dceaa64dcfdec6"
format_stamp: "Formatted at 2026-09-28 04:58:30 on dist-test-slave-m39l"
server_key: "e157d537d3be7ac11c6d1453b0222857"
server_key_iv: "f527b88608e76c40ac1697b7276aab1c"
server_key_version: "encryptionkey@0"
I20260928 04:58:30.158842 22531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:30.162858 22531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:30.165529 22531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:30.165670 22531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:30.170552 22531 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45505
I20260928 04:58:30.170635 22598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45505 every 8 connection(s)
I20260928 04:58:30.171036 22531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:30.172160 22599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:30.174723 22599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6: Bootstrap starting.
I20260928 04:58:30.175420 22599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:30.175736 22599 log.cc:824] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:30.176599 22599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6: No bootstrap required, opened a new log
I20260928 04:58:30.178352 22599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3e88e37d1f4912b6dceaa64dcfdec6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45505 } }
I20260928 04:58:30.178607 22599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:58:30.178638 22599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3e88e37d1f4912b6dceaa64dcfdec6, State: Initialized, Role: FOLLOWER
I20260928 04:58:30.178812 22599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3e88e37d1f4912b6dceaa64dcfdec6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45505 } }
I20260928 04:58:30.178884 22599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:58:30.178906 22599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:58:30.178975 22599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:30.179335 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 22531
I20260928 04:58:30.179483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:30.179687 392 external_mini_cluster.cc:1428] Setting key cb7dff1df99450eb36473e799a08027d
I20260928 04:58:30.179764 22599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3e88e37d1f4912b6dceaa64dcfdec6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45505 } }
I20260928 04:58:30.179908 22599 leader_election.cc:302] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d3e88e37d1f4912b6dceaa64dcfdec6; no voters:
I20260928 04:58:30.180085 22599 leader_election.cc:288] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:58:30.180171 22602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:30.180361 22602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6 [term 1 LEADER]: Becoming Leader. State: Replica: 8d3e88e37d1f4912b6dceaa64dcfdec6, State: Running, Role: LEADER
I20260928 04:58:30.180549 22599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:30.180532 22602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3e88e37d1f4912b6dceaa64dcfdec6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45505 } }
I20260928 04:58:30.180977 22602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d3e88e37d1f4912b6dceaa64dcfdec6. Latest consensus state: current_term: 1 leader_uuid: "8d3e88e37d1f4912b6dceaa64dcfdec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3e88e37d1f4912b6dceaa64dcfdec6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45505 } } }
I20260928 04:58:30.181211 22602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:30.181142 22604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d3e88e37d1f4912b6dceaa64dcfdec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3e88e37d1f4912b6dceaa64dcfdec6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45505 } } }
I20260928 04:58:30.181676 22604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:30.182044 22609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:30.182701 22609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:30.182859 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:30.180268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33857 (local address 127.0.98.62:45505)
0928 04:58:30.180551 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:30.180562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:30.180602 (+ 40us) server_negotiation.cc:407] Connection header received
0928 04:58:30.180730 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:30.180738 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:30.180838 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:30.181308 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:30.182140 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:30.182144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:30.182145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:30.182149 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:30.182215 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:30.182236 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:30.182316 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:30.182519 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 04:58:30.182573 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":72,"threads_started":1}
I20260928 04:58:30.184131 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:30.184568 22609 catalog_manager.cc:1347] Generated new cluster ID: c6b57bce9c8d4c22a8cbdf4309c9f01a
I20260928 04:58:30.184628 22609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:58:30.192184 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:30.186989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39769 (local address 127.0.98.62:45505)
0928 04:58:30.187042 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:30.187044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:30.190896 (+ 3852us) server_negotiation.cc:407] Connection header received
0928 04:58:30.190936 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:30.190937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:30.190945 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:30.191042 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:30.191861 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:30.191863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:30.191864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:58:30.191869 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:30.191890 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:58:30.191895 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:30.191991 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:30.192056 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 04:58:30.192077 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 04:58:30.207103 22609 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:30.207320 22609 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:30.215374 22609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d3e88e37d1f4912b6dceaa64dcfdec6: Generated new TSK 0
I20260928 04:58:30.215653 22609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:58:30.222182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 22531
2026-09-28T04:58:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1369 ms)
[----------] 18 tests from AuthTokenIssuingTest (24897 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:30 dist-test-slave-m39l krb5kdc[22630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:30 dist-test-slave-m39l krb5kdc[22630](info): set up 2 sockets
Sep 28 04:58:30 dist-test-slave-m39l krb5kdc[22630](info): commencing operation
krb5kdc: starting...
W20260928 04:58:31.390892 392 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:31 dist-test-slave-m39l krb5kdc[22630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:58:31.553933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45621
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45621,127.0.98.61:42365,127.0.98.60:41977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:31.663378 22643 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 04:58:31.663642 22643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:31.663662 22643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:31.663686 22643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:31.665287 22643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:31.665380 22643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:31.665442 22643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:31.665580 22643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:31.665660 22643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:31.668569 22643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45621,127.0.98.61:42365,127.0.98.60:41977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:31.669108 22643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:31.669726 22643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:31.676479 22651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:31.676671 22649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:31.676671 22648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:31.677316 22643 server_base.cc:1034] running on GCE node
I20260928 04:58:31.677774 22643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:31.678231 22643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:31.679575 22643 hybrid_clock.cc:648] HybridClock initialized: now 1790571511679550 us; error 44 us; skew 500 ppm
Sep 28 04:58:31 dist-test-slave-m39l krb5kdc[22630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:31.682899 22643 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:58:31.683652 22643 webserver.cc:460] Webserver started at http://127.0.98.62:41147/ using document root <none> and password file <none>
I20260928 04:58:31.683929 22643 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:31.683983 22643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:31.684090 22643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:31.685369 22643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "538c3a843573427fabab0b78121e9c43"
format_stamp: "Formatted at 2026-09-28 04:58:31 on dist-test-slave-m39l"
server_key: "9936d6531b9ddf51f09e55bfc24b2375"
server_key_iv: "d3f710614413800c9f8080cc09ff94ba"
server_key_version: "encryptionkey@0"
I20260928 04:58:31.685727 22643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "538c3a843573427fabab0b78121e9c43"
format_stamp: "Formatted at 2026-09-28 04:58:31 on dist-test-slave-m39l"
server_key: "9936d6531b9ddf51f09e55bfc24b2375"
server_key_iv: "d3f710614413800c9f8080cc09ff94ba"
server_key_version: "encryptionkey@0"
I20260928 04:58:31.687508 22643 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 04:58:31.688561 22658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:31.689095 22643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:58:31.689350 22643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "538c3a843573427fabab0b78121e9c43"
format_stamp: "Formatted at 2026-09-28 04:58:31 on dist-test-slave-m39l"
server_key: "9936d6531b9ddf51f09e55bfc24b2375"
server_key_iv: "d3f710614413800c9f8080cc09ff94ba"
server_key_version: "encryptionkey@0"
I20260928 04:58:31.689462 22643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:31.704870 22643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:31.708192 22643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:31.708415 22643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:31.713135 22643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45621
I20260928 04:58:31.713229 22710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45621 every 8 connection(s)
I20260928 04:58:31.713655 22643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:31.714917 22711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:31.716738 22711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:58:31.721477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 22643
I20260928 04:58:31.721616 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:31.721833 392 external_mini_cluster.cc:1428] Setting key b31cfc7931b7f57bdab47f95e861095f
Sep 28 04:58:31 dist-test-slave-m39l krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:58:31.728421 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.717578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.0.98.62:45621)
0928 04:58:31.717781 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:31.717785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:31.717910 (+ 125us) server_negotiation.cc:407] Connection header received
0928 04:58:31.718473 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:31.718475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:31.718547 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:31.718633 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:31.719254 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:31.720104 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:31.721113 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:31.721732 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:31.721908 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:31.724443 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:31.724450 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:31.724451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:31.724458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:31.726264 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:31.726911 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:31.726915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:31.726915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:31.727024 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:31.727442 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:31.727443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:31.727443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:31.727722 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:31.727812 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:31.728086 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 04:58:31.728214 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":101,"threads_started":1}
I20260928 04:58:31.728521 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.717504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45621 (local address 127.0.0.1:57980)
0928 04:58:31.717669 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:31.717724 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:31.718319 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:31.718686 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:31.718689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:31.718978 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:31.719156 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:31.719161 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:31.720213 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:31.720214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:31.721000 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:31.721002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:31.721958 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:31.721960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:31.722042 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:31.722741 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:31.722806 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:31.724300 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:31.726373 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:31.726383 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:31.726385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:31.726769 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:31.727109 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:31.727110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:31.727111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:31.727260 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:31.727813 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:31.727822 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:31.727912 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 04:58:31.728025 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 04:58:31.728329 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":103,"thread_start_us":68,"threads_started":1}
I20260928 04:58:31.729440 22711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:58:31.729926 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.729764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42365 (local address 127.0.0.1:51686)
0928 04:58:31.729836 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:31.729893 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 04:58:31.730098 22660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42365: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260928 04:58:31.730652 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.730546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42365 (local address 127.0.0.1:51694)
0928 04:58:31.730585 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:31.730617 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 04:58:31.730805 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42365: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42365: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 04:58:31.751551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42365
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45621,127.0.98.61:42365,127.0.98.60:41977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:58:31.780088 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } attempt: 1
I20260928 04:58:31.780723 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.780523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42365 (local address 127.0.0.1:51696)
0928 04:58:31.780609 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:31.780679 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 04:58:31.781188 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.781082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42365 (local address 127.0.0.1:51712)
0928 04:58:31.781132 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:31.781165 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 04:58:31.781299 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42365: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42365: connect: Connection refused (error 111)
W20260928 04:58:31.838449 22718 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 04:58:31.838732 22718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:31.838752 22718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:31.838775 22718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:31.839975 22718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:31.840023 22718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:31.840039 22718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:31.840051 22718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:31.840063 22718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:31.841722 22718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45621,127.0.98.61:42365,127.0.98.60:41977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:31.841974 22718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:31.842163 22718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:31.847805 22724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:31.847857 22723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:31.847890 22726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:31.848119 22718 server_base.cc:1034] running on GCE node
I20260928 04:58:31.848481 22718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:31.848860 22718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:31.849426 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } attempt: 2
I20260928 04:58:31.849934 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.849730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42365 (local address 127.0.0.1:51724)
0928 04:58:31.849813 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:31.849900 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 04:58:31.850068 22718 hybrid_clock.cc:648] HybridClock initialized: now 1790571511850053 us; error 29 us; skew 500 ppm
I20260928 04:58:31.850440 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.850269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42365 (local address 127.0.0.1:51730)
0928 04:58:31.850379 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:31.850412 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260928 04:58:31.850665 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42365: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42365: connect: Connection refused (error 111)
Sep 28 04:58:31 dist-test-slave-m39l krb5kdc[22630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:31.853211 22718 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 04:58:31.853719 22718 webserver.cc:460] Webserver started at http://127.0.98.61:38431/ using document root <none> and password file <none>
I20260928 04:58:31.853986 22718 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:31.854058 22718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:31.854214 22718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:31.855396 22718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-1/data/instance:
uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f"
format_stamp: "Formatted at 2026-09-28 04:58:31 on dist-test-slave-m39l"
server_key: "ce4ec6beceb33825c9ded30064165c3a"
server_key_iv: "10cc879cadd86509a11ef531ea5bf03b"
server_key_version: "encryptionkey@0"
I20260928 04:58:31.855775 22718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-1/wal/instance:
uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f"
format_stamp: "Formatted at 2026-09-28 04:58:31 on dist-test-slave-m39l"
server_key: "ce4ec6beceb33825c9ded30064165c3a"
server_key_iv: "10cc879cadd86509a11ef531ea5bf03b"
server_key_version: "encryptionkey@0"
I20260928 04:58:31.857342 22718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:58:31.858378 22733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:31.858661 22718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:58:31.858851 22718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-1/wal
uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f"
format_stamp: "Formatted at 2026-09-28 04:58:31 on dist-test-slave-m39l"
server_key: "ce4ec6beceb33825c9ded30064165c3a"
server_key_iv: "10cc879cadd86509a11ef531ea5bf03b"
server_key_version: "encryptionkey@0"
I20260928 04:58:31.858947 22718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:31.881907 22718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:31.884516 22718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:31.884709 22718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:31.889320 22718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42365
I20260928 04:58:31.889358 22785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42365 every 8 connection(s)
I20260928 04:58:31.889752 22718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-1/data/info.pb
I20260928 04:58:31.891036 22786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:31.892635 22786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:58:31.897491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 22718
I20260928 04:58:31.897637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-1/wal/instance
I20260928 04:58:31.897837 392 external_mini_cluster.cc:1428] Setting key e464ec94e499120fe3f4f92a4e3c7610
Sep 28 04:58:31 dist-test-slave-m39l krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:58:31.903702 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.893488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.0.98.62:45621)
0928 04:58:31.893556 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:31.893558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:31.893820 (+ 262us) server_negotiation.cc:407] Connection header received
0928 04:58:31.894370 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:31.894371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:31.894387 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:31.894450 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:31.895267 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:31.895844 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:31.896923 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:31.897142 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:31.897228 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:31.899676 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:31.899678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:31.899679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:31.899692 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:31.901639 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:31.902385 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:31.902386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:31.902386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:31.902445 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:31.902887 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:31.902888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:31.902889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:31.903017 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:31.903102 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:31.903488 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 04:58:31.903589 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:58:31.903904 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.893434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45621 (local address 127.0.0.1:57984)
0928 04:58:31.893678 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:31.893720 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:31.894252 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:31.894582 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:31.894585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:31.894926 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:31.895139 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:31.895144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:31.895949 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:31.895951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:31.896783 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:31.896785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:31.897252 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:31.897254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:31.897396 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:31.898003 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:31.898012 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:31.899533 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:31.901783 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:31.901798 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:31.901801 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:31.902203 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:31.902555 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:31.902556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:31.902556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:31.902751 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:31.903106 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:31.903114 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:31.903333 (+ 219us) client_negotiation.cc:769] Sending connection context
0928 04:58:31.903451 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 04:58:31.903639 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":117,"threads_started":1}
I20260928 04:58:31.904569 22786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 04:58:31 dist-test-slave-m39l krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:58:31.915776 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.904965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42365 (local address 127.0.0.1:51746)
0928 04:58:31.905228 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:31.905241 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:31.905302 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:31.905468 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:31.905469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:31.905643 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:31.905725 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:31.905726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:31.906786 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:31.906788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:31.907308 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:31.907310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:31.908133 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:31.908134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:31.908213 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:31.908757 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:31.908762 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:31.911837 (+ 3075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:31.914460 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:31.914461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:31.914462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:31.914696 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:31.914959 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:31.914960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:31.914960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:31.915024 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:31.915360 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:31.915361 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:31.915406 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 04:58:31.915498 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 04:58:31.915629 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
I20260928 04:58:31.915829 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.905051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51746 (local address 127.0.98.61:42365)
0928 04:58:31.905238 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:31.905242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:31.905295 (+ 53us) server_negotiation.cc:407] Connection header received
0928 04:58:31.905364 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:31.905365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:31.905378 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:31.905443 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:31.905852 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:31.906698 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:31.907408 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:31.907918 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:31.908133 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:31.911975 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:31.911993 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:31.911998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:31.912007 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:31.914334 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:31.914802 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:31.914807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:31.914807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:31.914873 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:31.915094 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:31.915095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:31.915095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:31.915281 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:31.915387 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:31.915546 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 04:58:31.915692 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":159,"thread_start_us":114,"threads_started":1}
I20260928 04:58:31.916846 22786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:58:31.917305 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.917122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41977 (local address 127.0.0.1:44320)
0928 04:58:31.917189 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:31.917270 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 04:58:31.917495 22735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
I20260928 04:58:31.918017 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.917898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41977 (local address 127.0.0.1:44334)
0928 04:58:31.917954 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:31.917990 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 04:58:31.918169 22786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41977: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 04:58:31.931062 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41977
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45621,127.0.98.61:42365,127.0.98.60:41977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:58:31.941847 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } attempt: 3
Sep 28 04:58:31 dist-test-slave-m39l krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:58:31.950192 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.942322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42365 (local address 127.0.0.1:51762)
0928 04:58:31.942392 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:31.942406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:31.942463 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:31.942651 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:31.942652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:31.942844 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:31.942919 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:31.942921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:31.943619 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:31.943620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:31.944175 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:31.944177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:31.944650 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:31.944651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:31.944734 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:31.945253 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:31.945258 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:31.946934 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:31.948986 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:31.948988 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:31.948989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:31.949196 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:31.949434 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:31.949435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:31.949435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:31.949506 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:31.949800 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:31.949801 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:31.949858 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 04:58:31.949958 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 04:58:31.950077 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 04:58:31.950192 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.942418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51762 (local address 127.0.98.61:42365)
0928 04:58:31.942510 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:31.942513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:31.942525 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:58:31.942555 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:31.942556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:31.942575 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:31.942644 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:31.943015 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:31.943521 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:31.944273 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:31.944566 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:31.944649 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:31.947043 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:31.947045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:31.947045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:31.947054 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:31.948803 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:31.949283 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:31.949285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:31.949285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:31.949355 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:31.949586 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:31.949588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:31.949588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:31.949725 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:31.949811 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:31.949985 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 04:58:31.950082 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 04:58:31.950663 22711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:58:31.951084 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.950944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41977 (local address 127.0.0.1:44340)
0928 04:58:31.951014 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:31.951055 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 04:58:31.951503 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.951406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41977 (local address 127.0.0.1:44344)
0928 04:58:31.951455 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:31.951483 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 04:58:31.951604 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41977: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
I20260928 04:58:31.967429 22786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } attempt: 1
I20260928 04:58:31.967944 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.967807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41977 (local address 127.0.0.1:44348)
0928 04:58:31.967873 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:31.967914 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 04:58:31.968317 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.968218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41977 (local address 127.0.0.1:44354)
0928 04:58:31.968265 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:31.968291 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 04:58:31.968400 22786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41977: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
I20260928 04:58:31.982762 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } attempt: 1
I20260928 04:58:31.983345 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.983162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41977 (local address 127.0.0.1:44368)
0928 04:58:31.983251 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:31.983307 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 04:58:31.983736 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:31.983615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41977 (local address 127.0.0.1:44384)
0928 04:58:31.983666 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:31.983707 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 04:58:31.983947 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41977: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
W20260928 04:58:32.022871 22793 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 04:58:32.023123 22793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:32.023144 22793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:32.023169 22793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:32.024329 22793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:32.024371 22793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:32.024389 22793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:32.024403 22793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:32.024417 22793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:32.026774 22793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45621,127.0.98.61:42365,127.0.98.60:41977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:32.027233 22793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:32.027522 22793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:32.033838 22798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:32.034144 22799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:32.034415 22801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:32.034907 22793 server_base.cc:1034] running on GCE node
I20260928 04:58:32.035355 22793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:32.035722 22793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:32.036608 22786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } attempt: 2
I20260928 04:58:32.036994 22793 hybrid_clock.cc:648] HybridClock initialized: now 1790571512036958 us; error 39 us; skew 500 ppm
I20260928 04:58:32.037045 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.036910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41977 (local address 127.0.0.1:44400)
0928 04:58:32.036970 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:32.037005 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 04:58:32.037482 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.037362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41977 (local address 127.0.0.1:44408)
0928 04:58:32.037411 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:32.037446 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 04:58:32.037688 22786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41977: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
Sep 28 04:58:32 dist-test-slave-m39l krb5kdc[22630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:32.041090 22793 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 04:58:32.041738 22793 webserver.cc:460] Webserver started at http://127.0.98.60:35547/ using document root <none> and password file <none>
I20260928 04:58:32.042052 22793 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:32.042131 22793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:32.042284 22793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:32.043753 22793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-2/data/instance:
uuid: "6e1b06a12d094c06987e58213db80039"
format_stamp: "Formatted at 2026-09-28 04:58:32 on dist-test-slave-m39l"
server_key: "02ab92e5877b00e9bf1c94a8f0fa4a07"
server_key_iv: "2c7d1901a1cd8d87b84d8ba5d8822fac"
server_key_version: "encryptionkey@0"
I20260928 04:58:32.044238 22793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-2/wal/instance:
uuid: "6e1b06a12d094c06987e58213db80039"
format_stamp: "Formatted at 2026-09-28 04:58:32 on dist-test-slave-m39l"
server_key: "02ab92e5877b00e9bf1c94a8f0fa4a07"
server_key_iv: "2c7d1901a1cd8d87b84d8ba5d8822fac"
server_key_version: "encryptionkey@0"
I20260928 04:58:32.046288 22793 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 04:58:32.047374 22808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:32.047828 22793 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:58:32.047966 22793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-2/wal
uuid: "6e1b06a12d094c06987e58213db80039"
format_stamp: "Formatted at 2026-09-28 04:58:32 on dist-test-slave-m39l"
server_key: "02ab92e5877b00e9bf1c94a8f0fa4a07"
server_key_iv: "2c7d1901a1cd8d87b84d8ba5d8822fac"
server_key_version: "encryptionkey@0"
I20260928 04:58:32.048071 22793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:32.059118 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } attempt: 2
I20260928 04:58:32.059679 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.059513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41977 (local address 127.0.0.1:44410)
0928 04:58:32.059596 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:32.059645 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 04:58:32.060127 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.059981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41977 (local address 127.0.0.1:44420)
0928 04:58:32.060047 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:32.060093 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 04:58:32.060297 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41977: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41977: connect: Connection refused (error 111)
I20260928 04:58:32.065433 22793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:32.068063 22793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:32.068245 22793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:32.073129 22793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41977
I20260928 04:58:32.073170 22860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41977 every 8 connection(s)
I20260928 04:58:32.073570 22793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-2/data/info.pb
I20260928 04:58:32.074882 22861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:32.076732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 22793
I20260928 04:58:32.076838 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790571208412331-392-0/minicluster-data/master-2/wal/instance
I20260928 04:58:32.077011 392 external_mini_cluster.cc:1428] Setting key 2881b8cfad512ac39536be82dad0602d
I20260928 04:58:32.076741 22861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:58:32 dist-test-slave-m39l krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 04:58:32 dist-test-slave-m39l krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:58:32.089023 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.077671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57992 (local address 127.0.98.62:45621)
0928 04:58:32.077724 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.077727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.077989 (+ 262us) server_negotiation.cc:407] Connection header received
0928 04:58:32.078638 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.078640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.078654 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.078735 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.079401 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.079906 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.081115 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.081428 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.081520 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:32.084692 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:32.084694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:32.084695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:32.084707 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:32.087138 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.087720 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.087721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.087721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.087781 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.088196 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.088197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.088197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.088368 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:32.088438 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:32.088826 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 04:58:32.088920 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 04:58:32.089401 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.077582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44434 (local address 127.0.98.60:41977)
0928 04:58:32.077816 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.077826 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.077860 (+ 34us) server_negotiation.cc:407] Connection header received
0928 04:58:32.078481 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.078492 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.078548 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.078644 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.079364 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.080473 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.081481 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.082200 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.082436 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:32.085568 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:32.085580 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:32.085581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:32.085591 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:32.087531 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.088028 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.088031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.088031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.088224 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.088573 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.088574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.088574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.088727 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:32.088936 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:32.089124 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 04:58:32.089218 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":93,"threads_started":1}
I20260928 04:58:32.089403 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.077611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45621 (local address 127.0.0.1:57992)
0928 04:58:32.077822 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:32.077867 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:32.078481 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:32.078771 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:32.078774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:32.079078 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:32.079259 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:32.079263 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.079993 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.079994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:32.081014 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:32.081016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.081580 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.081581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:32.081732 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:32.082637 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:32.082645 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:32.084566 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:32.087245 (+ 2679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:32.087253 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:32.087255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:32.087625 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:32.087854 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:32.087855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:32.087855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:32.088023 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:32.088466 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:32.088474 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:32.088655 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 04:58:32.088799 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 04:58:32.089151 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":97,"threads_started":1}
I20260928 04:58:32.090297 22861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:58:32 dist-test-slave-m39l krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 28 04:58:32 dist-test-slave-m39l krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:58:32.099210 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.090661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42365 (local address 127.0.0.1:51764)
0928 04:58:32.090701 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:32.090710 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:32.090741 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:32.090936 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:32.090938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:32.091171 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:32.091232 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:32.091235 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.092089 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.092090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:32.092769 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:32.092771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.093194 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.093195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:32.093271 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:32.093844 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:32.093849 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:32.095290 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:32.097803 (+ 2513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:32.097804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:32.097805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:32.098048 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:32.098293 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:32.098294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:32.098294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:32.098402 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:32.098822 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:32.098823 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:32.098885 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 04:58:32.098980 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 04:58:32.099107 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 04:58:32.099237 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.090749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51764 (local address 127.0.98.61:42365)
0928 04:58:32.090804 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.090807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.090814 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:58:32.090839 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.090840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.090851 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.090920 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.091338 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.091996 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.092877 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.093108 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.093195 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:32.095418 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:32.095420 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:32.095420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:32.095428 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:32.097692 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.098151 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.098152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.098153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.098214 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.098479 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.098480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.098480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.098726 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:32.098795 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:32.099003 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 04:58:32.099103 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 04:58:32.099707 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.090746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.0.98.62:45621)
0928 04:58:32.090793 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.090795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.090804 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:58:32.091036 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.091037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.091049 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.091104 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.091678 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.092325 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.093194 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.093494 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.093548 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:32.096019 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:32.096020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:32.096021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:32.096029 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:32.097918 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.098425 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.098427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.098427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.098534 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.098821 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.098822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.098823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.099005 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:32.099189 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:32.099230 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 04:58:32.099624 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 04:58:32.099707 22861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:58:32 dist-test-slave-m39l krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 04:58:32 dist-test-slave-m39l krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:58:32.108021 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.100098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41977 (local address 127.0.0.1:44444)
0928 04:58:32.100224 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:32.100237 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:32.100294 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:32.100426 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:32.100426 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:32.100621 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:32.100699 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:32.100701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.101538 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.101539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:32.102209 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:32.102211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.102769 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.102771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:32.102813 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:32.103337 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:32.103343 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:32.104882 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:32.106928 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:32.106929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:32.106930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:32.107091 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:32.107332 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:32.107332 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:32.107333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:32.107391 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:32.107663 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:32.107665 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:32.107706 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 04:58:32.107783 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 04:58:32.107903 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 04:58:32.108033 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.100181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.0.98.60:41977)
0928 04:58:32.100218 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.100219 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.100277 (+ 58us) server_negotiation.cc:407] Connection header received
0928 04:58:32.100360 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.100361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.100371 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.100424 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.100779 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.101444 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.102347 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.102658 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.102753 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:32.104978 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:32.104979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:32.104980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:32.104988 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:32.106834 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.107189 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.107190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.107190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.107247 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.107455 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.107456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.107456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.107588 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:32.107660 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:32.107821 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 04:58:32.107956 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":17}
I20260928 04:58:32.108707 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.100481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.0.98.61:42365)
0928 04:58:32.100518 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.100520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.100527 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:58:32.100545 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.100545 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.100554 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.100606 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.101201 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.101797 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.102720 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.103037 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.103117 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:32.105761 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:32.105763 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:32.105763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:32.105773 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:32.107486 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.107898 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.107899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.107900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.107974 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.108187 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.108188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.108189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.108327 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:32.108461 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:32.108580 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 04:58:32.108645 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:58:32.110402 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:32.110414 22861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039: Bootstrap starting.
I20260928 04:58:32.111464 22861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:32.111812 22861 log.cc:824] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:32.111441 22685 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "538c3a843573427fabab0b78121e9c43"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57994
I20260928 04:58:32.112710 22861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039: No bootstrap required, opened a new log
I20260928 04:58:32.114804 22861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c3a843573427fabab0b78121e9c43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } peers { permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } } peers { permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } }
I20260928 04:58:32.114982 22861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1b06a12d094c06987e58213db80039, State: Initialized, Role: FOLLOWER
I20260928 04:58:32.115065 22861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c3a843573427fabab0b78121e9c43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } peers { permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } } peers { permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } }
I20260928 04:58:32.115446 22861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:32.115366 22867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c3a843573427fabab0b78121e9c43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } peers { permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } } peers { permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } } }
I20260928 04:58:32.115520 22867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 04:58:32.117715 22878 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:58:32.117873 22878 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:58:32.117969 22793 master_runner.cc:186] Error getting master registration for 127.0.98.62:45621: OK, instance_id { permanent_uuid: "538c3a843573427fabab0b78121e9c43" instance_seqno: 1790571511711785 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 04:58:32.121160 22793 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 04:58:32.128865 22786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } attempt: 3
Sep 28 04:58:32 dist-test-slave-m39l krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:58:32.137693 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.129320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41977 (local address 127.0.0.1:44446)
0928 04:58:32.129390 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:32.129403 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:32.129457 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:32.129765 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:32.129766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:32.129971 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:32.130031 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:32.130033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.131031 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.131032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:32.131870 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:32.131873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.132425 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.132426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:32.132483 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:32.133022 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:32.133028 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:32.134722 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:32.136487 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:32.136489 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:32.136489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:32.136677 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:32.136938 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:32.136939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:32.136940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:32.137004 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:32.137329 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:32.137330 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:32.137379 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 04:58:32.137471 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 04:58:32.137594 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 04:58:32.137693 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.129522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44446 (local address 127.0.98.60:41977)
0928 04:58:32.129595 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.129598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.129611 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:58:32.129641 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.129642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.129654 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.129733 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.130122 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.130932 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.132005 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.132322 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.132417 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:32.134815 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:32.134816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:32.134817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:32.134839 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:32.136381 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.136775 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.136776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.136777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.136848 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.137085 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.137085 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.137086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.137257 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:32.137338 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:32.137497 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 04:58:32.137590 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:58:32.139182 22786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f: Bootstrap starting.
I20260928 04:58:32.140033 22786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:32.140383 22786 log.cc:824] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:32.141216 22786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f: No bootstrap required, opened a new log
I20260928 04:58:32.143412 22786 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c3a843573427fabab0b78121e9c43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } peers { permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } } peers { permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } }
I20260928 04:58:32.143589 22786 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2dbaef2b6ef45d6bd72be2c6b84669f, State: Initialized, Role: FOLLOWER
I20260928 04:58:32.143669 22786 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c3a843573427fabab0b78121e9c43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } peers { permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } } peers { permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } }
I20260928 04:58:32.143987 22881 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c3a843573427fabab0b78121e9c43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } peers { permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } } peers { permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } } }
I20260928 04:58:32.144062 22786 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:32.144078 22881 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f [sys.catalog]: This master's current role is: FOLLOWER
W20260928 04:58:32.146049 22892 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:58:32.146116 22892 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:58:32.146391 22718 master_runner.cc:186] Error getting master registration for 127.0.98.62:45621: OK, instance_id { permanent_uuid: "538c3a843573427fabab0b78121e9c43" instance_seqno: 1790571511711785 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 04:58:32.152288 22718 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 04:58:32.159477 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } attempt: 3
Sep 28 04:58:32 dist-test-slave-m39l krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:58:32.168457 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.159970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44448 (local address 127.0.98.60:41977)
0928 04:58:32.160018 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.160021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.160032 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:58:32.160064 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.160065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.160077 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.160158 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.160513 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.161309 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.162407 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.162773 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.162855 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:32.165237 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:32.165238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:32.165239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:32.165248 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:32.167140 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.167607 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.167608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.167608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.167680 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.167882 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.167884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.167884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.168038 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:32.168103 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:32.168262 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 04:58:32.168356 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:58:32.168457 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.159878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41977 (local address 127.0.0.1:44448)
0928 04:58:32.159950 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:32.159961 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:32.160005 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:32.160178 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:32.160179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:32.160387 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:32.160444 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:32.160446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.161419 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.161420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:32.162306 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:32.162310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.162862 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.162864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:32.162930 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:32.163473 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:32.163479 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:32.165134 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:32.167253 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:32.167255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:32.167255 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:32.167519 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:32.167744 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:32.167745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:32.167745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:32.167815 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:32.168099 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:32.168100 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:32.168155 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 04:58:32.168220 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 04:58:32.168339 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 04:58:32.170464 22711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43: Bootstrap starting.
I20260928 04:58:32.171283 22711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:32.171589 22711 log.cc:824] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:32.172456 22711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43: No bootstrap required, opened a new log
I20260928 04:58:32.173851 22711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c3a843573427fabab0b78121e9c43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } peers { permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } } peers { permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } }
I20260928 04:58:32.173990 22711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538c3a843573427fabab0b78121e9c43, State: Initialized, Role: FOLLOWER
I20260928 04:58:32.174067 22711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c3a843573427fabab0b78121e9c43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } peers { permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } } peers { permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } }
I20260928 04:58:32.174366 22711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:32.174476 22685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 04:58:32.174584 22685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:32.175524 22685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c3a843573427fabab0b78121e9c43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } peers { permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } } peers { permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } }
I20260928 04:58:32.176162 22760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "538c3a843573427fabab0b78121e9c43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f"
I20260928 04:58:32.176337 22760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:32.177294 22760 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 538c3a843573427fabab0b78121e9c43 in term 1.
I20260928 04:58:32.177666 22660 leader_election.cc:302] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 538c3a843573427fabab0b78121e9c43, b2dbaef2b6ef45d6bd72be2c6b84669f; no voters:
I20260928 04:58:32.178171 22685 leader_election.cc:288] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [CANDIDATE]: Term 1 election: Requested vote from peers b2dbaef2b6ef45d6bd72be2c6b84669f (127.0.98.61:42365), 6e1b06a12d094c06987e58213db80039 (127.0.98.60:41977)
I20260928 04:58:32.178264 22895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c3a843573427fabab0b78121e9c43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } peers { permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } } peers { permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } } }
I20260928 04:58:32.178354 22895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:58:32.178404 22895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:32.178448 22895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [term 1 LEADER]: Becoming Leader. State: Replica: 538c3a843573427fabab0b78121e9c43, State: Running, Role: LEADER
I20260928 04:58:32.178548 22895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c3a843573427fabab0b78121e9c43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } peers { permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } } peers { permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } }
I20260928 04:58:32.178716 22835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "538c3a843573427fabab0b78121e9c43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6e1b06a12d094c06987e58213db80039"
I20260928 04:58:32.178843 22835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:32.178968 22895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 538c3a843573427fabab0b78121e9c43. Latest consensus state: current_term: 1 leader_uuid: "538c3a843573427fabab0b78121e9c43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c3a843573427fabab0b78121e9c43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } peers { permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } } peers { permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } } }
I20260928 04:58:32.179036 22895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [sys.catalog]: This master's current role is: LEADER
W20260928 04:58:32.179867 22906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:58:32.179911 22906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:58:32.180042 22909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:32.180186 22909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:32.180179 22835 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 538c3a843573427fabab0b78121e9c43 in term 1.
I20260928 04:58:32.183934 22643 master_runner.cc:263] No leader master found from master 538c3a843573427fabab0b78121e9c43
I20260928 04:58:32.184016 22643 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 04:58:32.189739 22835 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039 [term 1 FOLLOWER]: Refusing update from remote peer 538c3a843573427fabab0b78121e9c43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:58:32.190088 22907 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:58:32.191630 22867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 538c3a843573427fabab0b78121e9c43. Latest consensus state: current_term: 1 leader_uuid: "538c3a843573427fabab0b78121e9c43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c3a843573427fabab0b78121e9c43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } peers { permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } } peers { permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } } }
I20260928 04:58:32.191725 22867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:58:32.192121 22760 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f [term 1 FOLLOWER]: Refusing update from remote peer 538c3a843573427fabab0b78121e9c43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:58:32.192390 22907 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:58:32.193709 22881 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 538c3a843573427fabab0b78121e9c43. Latest consensus state: current_term: 1 leader_uuid: "538c3a843573427fabab0b78121e9c43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c3a843573427fabab0b78121e9c43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } peers { permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } } peers { permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } } }
I20260928 04:58:32.193804 22881 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:58:32.194005 22907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "538c3a843573427fabab0b78121e9c43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c3a843573427fabab0b78121e9c43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } peers { permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } } peers { permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } } }
I20260928 04:58:32.194113 22907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:32.194445 22867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "538c3a843573427fabab0b78121e9c43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c3a843573427fabab0b78121e9c43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } peers { permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } } peers { permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } } }
I20260928 04:58:32.194576 22867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:58:32.194823 22911 mvcc.cc:205] Tried to move back new op lower bound from 7334180913926025216 to 7334180913884008448. Current Snapshot: MvccSnapshot[applied={T|T < 7334180913926025216}]
I20260928 04:58:32.195365 22919 mvcc.cc:205] Tried to move back new op lower bound from 7334180913926025216 to 7334180913884008448. Current Snapshot: MvccSnapshot[applied={T|T < 7334180913926025216}]
I20260928 04:58:32.195745 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.184444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58008 (local address 127.0.98.62:45621)
0928 04:58:32.184490 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.184492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.185362 (+ 870us) server_negotiation.cc:407] Connection header received
0928 04:58:32.185475 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.185476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.185487 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.185517 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.186379 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.187174 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.188495 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.188801 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.188845 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:32.190884 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:32.190886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:32.190887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:32.190894 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:32.192964 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.194386 (+ 1422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.194388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.194388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.194448 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.195029 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.195031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.195031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.195188 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:32.195268 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:32.195576 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 04:58:32.195655 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:58:32.197268 22895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "538c3a843573427fabab0b78121e9c43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c3a843573427fabab0b78121e9c43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } peers { permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } } peers { permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } } }
I20260928 04:58:32.197395 22895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43 [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:32.197824 22881 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "538c3a843573427fabab0b78121e9c43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c3a843573427fabab0b78121e9c43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } peers { permanent_uuid: "b2dbaef2b6ef45d6bd72be2c6b84669f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42365 } } peers { permanent_uuid: "6e1b06a12d094c06987e58213db80039" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41977 } } }
I20260928 04:58:32.197928 22881 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:58:32.198736 22921 mvcc.cc:205] Tried to move back new op lower bound from 7334180913926025216 to 7334180913884008448. Current Snapshot: MvccSnapshot[applied={T|T < 7334180913926025216}]
I20260928 04:58:32.201898 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.184807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.0.98.61:42365)
0928 04:58:32.184848 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.184850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.190666 (+ 5816us) server_negotiation.cc:407] Connection header received
0928 04:58:32.190786 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.190787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.190849 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.190885 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.191568 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.192253 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.194584 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.194897 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.194946 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:32.198589 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:32.198591 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:32.198591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:32.198599 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:32.200491 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.201000 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.201001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.201001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.201064 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.201257 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.201258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.201258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.201397 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:32.201493 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:32.201700 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 04:58:32.201800 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:58:32.202069 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:32.185092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44454 (local address 127.0.98.60:41977)
0928 04:58:32.185130 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.185132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.194653 (+ 9521us) server_negotiation.cc:407] Connection header received
0928 04:58:32.194755 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.194756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.194769 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.194800 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.195463 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.196168 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.197131 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:32.197391 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:32.197438 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:32.199024 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:32.199025 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:32.199026 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:32.199034 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:32.200863 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.201243 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.201244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.201244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.201300 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:32.201509 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:32.201510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:32.201510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:32.201635 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:32.201701 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:32.201896 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 04:58:32.201975 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:58:32.203120 22909 catalog_manager.cc:1347] Generated new cluster ID: 7e52ba4bcc514fe593b732e8ec15a40e
I20260928 04:58:32.203186 22909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:58:32.233799 22909 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:32.234653 22909 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:32.239503 22909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 538c3a843573427fabab0b78121e9c43: Generated new TSK 0
I20260928 04:58:32.239804 22909 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 04:58:32.268287 22714 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:32.266563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.0.98.62:45621)
0928 04:58:32.266611 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.266614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.267516 (+ 902us) server_negotiation.cc:407] Connection header received
0928 04:58:32.267639 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.267640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.267652 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.267686 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.268195 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58014: BlockingRecv error: recv got EOF from 127.0.0.1:58014 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 04:58:32.269163 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:32.266945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.0.98.61:42365)
0928 04:58:32.266988 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.266989 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.268607 (+ 1618us) server_negotiation.cc:407] Connection header received
0928 04:58:32.268727 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.268728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.268739 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.268769 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.269096 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51792: BlockingRecv error: recv got EOF from 127.0.0.1:51792 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 04:58:32.269917 22864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:32.267230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.0.98.60:41977)
0928 04:58:32.267272 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.267273 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.269405 (+ 2132us) server_negotiation.cc:407] Connection header received
0928 04:58:32.269500 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.269501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.269512 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.269541 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.269849 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44470: BlockingRecv error: recv got EOF from 127.0.0.1:44470 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 04:58:32.272653 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:32.271820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.0.98.61:42365)
0928 04:58:32.271862 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.271864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.272101 (+ 237us) server_negotiation.cc:407] Connection header received
0928 04:58:32.272211 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.272212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.272232 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.272261 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.272597 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51798: BlockingRecv error: recv got EOF from 127.0.0.1:51798 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 04:58:32.280068 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:32.278968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51814 (local address 127.0.98.61:42365)
0928 04:58:32.279019 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.279021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.279375 (+ 354us) server_negotiation.cc:407] Connection header received
0928 04:58:32.279491 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.279492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.279506 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.279538 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.280004 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51814: BlockingRecv error: recv got EOF from 127.0.0.1:51814 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 04:58:32.284711 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:32.283608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.0.98.61:42365)
0928 04:58:32.283672 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.283674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.284036 (+ 362us) server_negotiation.cc:407] Connection header received
0928 04:58:32.284146 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.284147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.284160 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.284192 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.284623 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51820: BlockingRecv error: recv got EOF from 127.0.0.1:51820 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 04:58:32.291361 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:32.290292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51824 (local address 127.0.98.61:42365)
0928 04:58:32.290340 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.290342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.290714 (+ 372us) server_negotiation.cc:407] Connection header received
0928 04:58:32.290824 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.290826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.290839 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.290870 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.291295 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51824: BlockingRecv error: recv got EOF from 127.0.0.1:51824 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 04:58:32.301745 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:32.300802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51838 (local address 127.0.98.61:42365)
0928 04:58:32.300852 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.300854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.301116 (+ 262us) server_negotiation.cc:407] Connection header received
0928 04:58:32.301219 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.301220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.301233 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.301263 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.301682 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51838: BlockingRecv error: recv got EOF from 127.0.0.1:51838 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 04:58:32.320294 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:32.319488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51854 (local address 127.0.98.61:42365)
0928 04:58:32.319554 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.319556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.319674 (+ 118us) server_negotiation.cc:407] Connection header received
0928 04:58:32.319761 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.319762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.319775 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.319859 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.320219 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51854: BlockingRecv error: recv got EOF from 127.0.0.1:51854 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 04:58:32.354223 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:32.353415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.0.98.61:42365)
0928 04:58:32.353493 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.353496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.353630 (+ 134us) server_negotiation.cc:407] Connection header received
0928 04:58:32.353706 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.353707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.353720 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.353777 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.354147 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51862: BlockingRecv error: recv got EOF from 127.0.0.1:51862 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 04:58:32.420347 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:32.419549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.0.98.61:42365)
0928 04:58:32.419642 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.419644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.419663 (+ 19us) server_negotiation.cc:407] Connection header received
0928 04:58:32.419747 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.419748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.419761 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.419815 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.420262 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51868: BlockingRecv error: recv got EOF from 127.0.0.1:51868 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 04:58:32.550442 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:32.549675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.0.98.61:42365)
0928 04:58:32.549775 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.549777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.549789 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:58:32.549877 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.549878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.549890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.549960 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.550374 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51878: BlockingRecv error: recv got EOF from 127.0.0.1:51878 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 04:58:32.808611 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:32.807719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.0.98.61:42365)
0928 04:58:32.807900 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:32.807902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:32.807915 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:58:32.807957 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:32.807957 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:32.807970 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:32.808045 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:32.808509 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51880: BlockingRecv error: recv got EOF from 127.0.0.1:51880 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
I20260928 04:58:33.118391 22878 catalog_manager.cc:1259] Loaded cluster ID: 7e52ba4bcc514fe593b732e8ec15a40e
I20260928 04:58:33.118520 22878 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039: loading cluster ID for follower catalog manager: success
I20260928 04:58:33.119374 22878 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039: acquiring CA information for follower catalog manager: success
I20260928 04:58:33.119707 22878 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6e1b06a12d094c06987e58213db80039: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:58:33.146638 22892 catalog_manager.cc:1259] Loaded cluster ID: 7e52ba4bcc514fe593b732e8ec15a40e
I20260928 04:58:33.146724 22892 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f: loading cluster ID for follower catalog manager: success
I20260928 04:58:33.147761 22892 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f: acquiring CA information for follower catalog manager: success
I20260928 04:58:33.148124 22892 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b2dbaef2b6ef45d6bd72be2c6b84669f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:58:33.324681 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:33.321917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.0.98.61:42365)
0928 04:58:33.322055 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:33.322058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:33.322069 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:58:33.322137 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:33.322138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:33.322211 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:33.322295 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:58:33.322506 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:33.323016 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:33.323819 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:33.324035 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:33.324115 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:33.324307 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:58:33.324427 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:58:33.324524 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:33.324549 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 04:58:33.324573 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":57,"threads_started":1}
I20260928 04:58:33.329478 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:33.326856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44482 (local address 127.0.98.60:41977)
0928 04:58:33.326990 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:33.326992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:33.327002 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:58:33.327104 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:33.327104 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:33.327148 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:33.327237 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:58:33.327342 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:33.327844 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:33.328630 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:33.328846 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:33.328925 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:33.329104 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:58:33.329224 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:58:33.329327 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:33.329348 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 04:58:33.329371 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260928 04:58:33.335179 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:33.331911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.0.98.61:42365)
0928 04:58:33.331957 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:33.331959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:33.332046 (+ 87us) server_negotiation.cc:407] Connection header received
0928 04:58:33.332203 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:33.332205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:33.332220 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:33.332321 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:58:33.332487 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:33.333369 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:33.334354 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:33.334642 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:33.334701 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:33.334873 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:58:33.334914 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:58:33.334973 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:33.335056 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 04:58:33.335082 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:58:33.335438 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:33.332133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44490 (local address 127.0.98.60:41977)
0928 04:58:33.332185 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:33.332188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:33.332543 (+ 355us) server_negotiation.cc:407] Connection header received
0928 04:58:33.332605 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:33.332606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:33.332618 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:33.332669 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:58:33.332812 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:33.333575 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:33.334531 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:33.334846 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:33.334912 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:33.335085 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:58:33.335118 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:58:33.335179 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:33.335326 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 04:58:33.335358 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:58:33.335587 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:33.331756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.0.98.62:45621)
0928 04:58:33.331925 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:33.331928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:33.332049 (+ 121us) server_negotiation.cc:407] Connection header received
0928 04:58:33.332140 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:33.332141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:33.332199 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:33.332294 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:58:33.332451 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:33.333120 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:33.334192 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:33.334579 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:33.334654 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:33.335058 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:58:33.335234 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:58:33.335301 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:33.335469 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 04:58:33.335498 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":109,"threads_started":1}
I20260928 04:58:33.337739 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 22643
I20260928 04:58:33.343262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 22718
I20260928 04:58:33.351459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 22793
2026-09-28T04:58:33Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3131 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:58:33 dist-test-slave-m39l krb5kdc[23008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:58:33 dist-test-slave-m39l krb5kdc[23008](info): set up 2 sockets
Sep 28 04:58:33 dist-test-slave-m39l krb5kdc[23008](info): commencing operation
krb5kdc: starting...
W20260928 04:58:34.423285 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/test-user.keytab.
Sep 28 04:58:34 dist-test-slave-m39l krb5kdc[23008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:58:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:58:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:58:34.562786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37995
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37995,127.0.98.61:44347,127.0.98.60:39179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:58:34.654278 23021 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 04:58:34.654652 23021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:34.654696 23021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:34.654744 23021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:34.656036 23021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:34.656116 23021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:34.656142 23021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:34.656177 23021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:34.656203 23021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:34.657931 23021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37995,127.0.98.61:44347,127.0.98.60:39179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:34.658264 23021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:34.658589 23021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:34.664024 23027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:34.664062 23026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:34.664268 23021 server_base.cc:1034] running on GCE node
W20260928 04:58:34.664040 23029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:34.664608 23021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:34.664917 23021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:34.666095 23021 hybrid_clock.cc:648] HybridClock initialized: now 1790571514666089 us; error 36 us; skew 500 ppm
Sep 28 04:58:34 dist-test-slave-m39l krb5kdc[23008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:34.676556 23021 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:58:34.677165 23021 webserver.cc:460] Webserver started at http://127.0.98.62:45659/ using document root <none> and password file <none>
I20260928 04:58:34.677426 23021 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:34.677503 23021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:34.677664 23021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:34.679260 23021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-0/data/instance:
uuid: "9abace7644b640cd8501184b3fc02fad"
format_stamp: "Formatted at 2026-09-28 04:58:34 on dist-test-slave-m39l"
server_key: "bb5cb3d819fa4d4e8f31a4a35980b452"
server_key_iv: "3409d6c76d99825d29007f1f178acdb4"
server_key_version: "encryptionkey@0"
I20260928 04:58:34.679785 23021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-0/wal/instance:
uuid: "9abace7644b640cd8501184b3fc02fad"
format_stamp: "Formatted at 2026-09-28 04:58:34 on dist-test-slave-m39l"
server_key: "bb5cb3d819fa4d4e8f31a4a35980b452"
server_key_iv: "3409d6c76d99825d29007f1f178acdb4"
server_key_version: "encryptionkey@0"
I20260928 04:58:34.681650 23021 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 04:58:34.682749 23036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:34.682917 23021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:34.683040 23021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-0/wal
uuid: "9abace7644b640cd8501184b3fc02fad"
format_stamp: "Formatted at 2026-09-28 04:58:34 on dist-test-slave-m39l"
server_key: "bb5cb3d819fa4d4e8f31a4a35980b452"
server_key_iv: "3409d6c76d99825d29007f1f178acdb4"
server_key_version: "encryptionkey@0"
I20260928 04:58:34.683127 23021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:34.705616 23021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:34.708235 23021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:34.708364 23021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:34.712601 23021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37995
I20260928 04:58:34.712675 23088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37995 every 8 connection(s)
I20260928 04:58:34.713061 23021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:58:34.714287 23089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:34.716154 23089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:58:34.720285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 23021
I20260928 04:58:34.720419 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-0/wal/instance
I20260928 04:58:34.720649 392 external_mini_cluster.cc:1428] Setting key 917699f233d06764a51b8e8973aa9e78
Sep 28 04:58:34 dist-test-slave-m39l krb5kdc[23008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260928 04:58:34.729353 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.717005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37995 (local address 127.0.0.1:44952)
0928 04:58:34.717225 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:34.717278 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:34.717978 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:34.718798 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:34.718802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:34.719108 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:34.719308 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:34.719313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:34.720993 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:34.720995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:34.721739 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:34.721741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:34.722516 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:34.722517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:34.722608 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:34.723316 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:34.723327 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:34.725332 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:34.727523 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:34.727531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:34.727533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:34.727764 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:34.728075 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:34.728076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:34.728076 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:34.728174 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:34.728644 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:34.728650 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:34.728727 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 04:58:34.728842 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 04:58:34.729125 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":111,"thread_start_us":119,"threads_started":1}
I20260928 04:58:34.729353 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.717155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.0.98.62:37995)
0928 04:58:34.717315 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:34.717320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:34.717446 (+ 126us) server_negotiation.cc:407] Connection header received
0928 04:58:34.718207 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:34.718210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:34.718238 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:34.718453 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:34.719463 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:34.720869 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:34.721855 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:34.722363 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:34.722469 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:34.725450 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:34.725460 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:34.725460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:34.725469 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:34.727386 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:34.727882 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:34.727886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:34.727887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:34.727990 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:34.728250 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:34.728251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:34.728251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:34.728515 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:34.728627 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:34.728871 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 04:58:34.729051 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260928 04:58:34.730376 23089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:58:34.730897 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.730736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44347 (local address 127.0.0.1:40816)
0928 04:58:34.730798 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:34.730860 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 04:58:34.731083 23037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44347: connect: Connection refused (error 111)
I20260928 04:58:34.731614 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.731494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44347 (local address 127.0.0.1:40830)
0928 04:58:34.731543 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:34.731576 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 04:58:34.731745 23089 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44347: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44347: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 04:58:34.750989 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44347
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37995,127.0.98.61:44347,127.0.98.60:39179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:58:34.781015 23089 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } attempt: 1
I20260928 04:58:34.781636 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.781487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44347 (local address 127.0.0.1:40834)
0928 04:58:34.781559 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:34.781607 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 04:58:34.782029 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.781943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44347 (local address 127.0.0.1:40844)
0928 04:58:34.781989 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:34.782014 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 04:58:34.782214 23089 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44347: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44347: connect: Connection refused (error 111)
W20260928 04:58:34.838479 23096 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 04:58:34.838805 23096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:34.838825 23096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:34.838851 23096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:34.840404 23096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:34.840471 23096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:34.840487 23096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:34.840499 23096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:34.840512 23096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:34.842321 23096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37995,127.0.98.61:44347,127.0.98.60:39179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:34.842665 23096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:34.842952 23096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:34.848335 23102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:34.848340 23104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:34.848340 23101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:34.848474 23096 server_base.cc:1034] running on GCE node
I20260928 04:58:34.848913 23096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:34.849247 23096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:34.850435 23096 hybrid_clock.cc:648] HybridClock initialized: now 1790571514850422 us; error 38 us; skew 500 ppm
I20260928 04:58:34.850401 23089 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } attempt: 2
I20260928 04:58:34.850955 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.850771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44347 (local address 127.0.0.1:40854)
0928 04:58:34.850884 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:34.850922 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 04:58:34.851369 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.851260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44347 (local address 127.0.0.1:40868)
0928 04:58:34.851310 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:34.851338 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 04:58:34.851488 23089 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44347: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44347: connect: Connection refused (error 111)
Sep 28 04:58:34 dist-test-slave-m39l krb5kdc[23008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:34.853518 23096 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 04:58:34.854058 23096 webserver.cc:460] Webserver started at http://127.0.98.61:46655/ using document root <none> and password file <none>
I20260928 04:58:34.854307 23096 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:34.854358 23096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:34.854554 23096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:34.855842 23096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-1/data/instance:
uuid: "c02510c1bb304159b77156b7ac46eec1"
format_stamp: "Formatted at 2026-09-28 04:58:34 on dist-test-slave-m39l"
server_key: "0e0c1622102b71e323f4855a99b0388d"
server_key_iv: "8b07dd41670dfb02b0e9f6d3648f4c0f"
server_key_version: "encryptionkey@0"
I20260928 04:58:34.856274 23096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-1/wal/instance:
uuid: "c02510c1bb304159b77156b7ac46eec1"
format_stamp: "Formatted at 2026-09-28 04:58:34 on dist-test-slave-m39l"
server_key: "0e0c1622102b71e323f4855a99b0388d"
server_key_iv: "8b07dd41670dfb02b0e9f6d3648f4c0f"
server_key_version: "encryptionkey@0"
I20260928 04:58:34.857846 23096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:58:34.858919 23111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:34.859148 23096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:58:34.859272 23096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-1/wal
uuid: "c02510c1bb304159b77156b7ac46eec1"
format_stamp: "Formatted at 2026-09-28 04:58:34 on dist-test-slave-m39l"
server_key: "0e0c1622102b71e323f4855a99b0388d"
server_key_iv: "8b07dd41670dfb02b0e9f6d3648f4c0f"
server_key_version: "encryptionkey@0"
I20260928 04:58:34.859359 23096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:34.879105 23096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:34.881692 23096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:34.882212 23096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:34.887360 23096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44347
I20260928 04:58:34.887401 23163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44347 every 8 connection(s)
I20260928 04:58:34.887810 23096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-1/data/info.pb
I20260928 04:58:34.888841 23164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:34.891079 23164 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:58:34.896874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 23096
I20260928 04:58:34.897001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-1/wal/instance
I20260928 04:58:34.897238 392 external_mini_cluster.cc:1428] Setting key 24263c083a015bc909deaf70b39a12a7
Sep 28 04:58:34 dist-test-slave-m39l krb5kdc[23008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:58:34.902122 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.891919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44968 (local address 127.0.98.62:37995)
0928 04:58:34.891988 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:34.891990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:34.892282 (+ 292us) server_negotiation.cc:407] Connection header received
0928 04:58:34.892855 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:34.892856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:34.892931 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:34.892998 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:34.893710 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:34.894235 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:34.895380 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:34.895623 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:34.895709 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:34.898532 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:34.898534 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:34.898535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:34.898544 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:34.900371 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:34.900921 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:34.900923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:34.900923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:34.900995 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:34.901269 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:34.901269 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:34.901270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:34.901404 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:34.901480 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:34.901813 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 04:58:34.902008 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:58:34.902354 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.891903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37995 (local address 127.0.0.1:44968)
0928 04:58:34.892135 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:34.892177 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:34.892730 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:34.893097 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:34.893101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:34.893393 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:34.893610 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:34.893615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:34.894341 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:34.894342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:34.895255 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:34.895258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:34.895736 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:34.895737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:34.895894 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:34.896493 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:34.896503 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:34.898370 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:34.900533 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:34.900544 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:34.900547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:34.900817 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:34.901067 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:34.901068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:34.901069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:34.901191 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:34.901482 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:34.901491 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:34.901664 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 04:58:34.901784 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 04:58:34.902079 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":106,"threads_started":1}
I20260928 04:58:34.903259 23164 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 04:58:34 dist-test-slave-m39l krb5kdc[23008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:58:34.913061 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.903730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44347 (local address 127.0.0.1:40882)
0928 04:58:34.903994 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:34.904006 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:34.904069 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:34.904235 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:34.904236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:34.904406 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:34.904473 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:34.904475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:34.905457 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:34.905458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:34.906091 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:34.906093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:34.906821 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:34.906822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:34.906888 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:34.907359 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:34.907363 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:34.909731 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:34.911888 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:34.911889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:34.911890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:34.912084 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:34.912323 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:34.912324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:34.912324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:34.912389 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:34.912712 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:34.912713 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:34.912753 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 04:58:34.912815 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 04:58:34.912934 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247}
I20260928 04:58:34.913143 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.903807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40882 (local address 127.0.98.61:44347)
0928 04:58:34.903995 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:34.903999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:34.904061 (+ 62us) server_negotiation.cc:407] Connection header received
0928 04:58:34.904129 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:34.904131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:34.904142 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:34.904213 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:34.904556 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:34.905355 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:34.906187 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:34.906622 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:34.906687 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:34.909852 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:34.909870 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:34.909871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:34.909883 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:34.911757 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:34.912188 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:34.912193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:34.912193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:34.912253 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:34.912463 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:34.912464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:34.912464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:34.912639 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:34.912737 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:34.912855 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 04:58:34.913012 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":158,"thread_start_us":125,"threads_started":1}
I20260928 04:58:34.913970 23164 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:58:34.914397 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.914220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.0.1:33254)
0928 04:58:34.914280 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:34.914357 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 04:58:34.914584 23113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39179: connect: Connection refused (error 111)
I20260928 04:58:34.915174 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.915043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.0.1:33266)
0928 04:58:34.915114 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:34.915148 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 04:58:34.915347 23164 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39179: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39179: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 04:58:34.931463 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwNdUqW/build/release/bin/kudu
/tmp/dist-test-taskwNdUqW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39179
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37995,127.0.98.61:44347,127.0.98.60:39179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:58:34.942629 23089 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } attempt: 3
Sep 28 04:58:34 dist-test-slave-m39l krb5kdc[23008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:58:34.951637 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.943127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40888 (local address 127.0.98.61:44347)
0928 04:58:34.943204 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:34.943207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:34.943218 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:58:34.943268 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:34.943270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:34.943284 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:34.943354 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:34.943749 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:34.944266 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:34.945026 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:34.945253 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:34.945333 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:34.947372 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:34.947374 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:34.947374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:34.947382 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:34.950275 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:34.950767 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:34.950768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:34.950769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:34.950839 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:34.951061 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:34.951062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:34.951062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:34.951193 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:34.951265 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:34.951443 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 04:58:34.951533 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 04:58:34.951637 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.943053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44347 (local address 127.0.0.1:40888)
0928 04:58:34.943126 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:34.943136 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:34.943204 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:34.943377 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:34.943378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:34.943554 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:34.943645 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:34.943647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:34.944374 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:34.944376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:34.944927 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:34.944929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:34.945343 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:34.945344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:34.945437 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:34.945988 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:34.945993 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:34.947255 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:34.950402 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:34.950404 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:34.950404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:34.950687 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:34.950910 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:34.950911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:34.950912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:34.950984 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:34.951268 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:34.951269 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:34.951321 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 04:58:34.951404 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 04:58:34.951531 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 04:58:34.952291 23089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:58:34.952780 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.952623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.0.1:33274)
0928 04:58:34.952690 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:34.952727 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 04:58:34.953217 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.953087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.0.1:33276)
0928 04:58:34.953128 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:34.953187 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 04:58:34.953337 23089 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39179: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39179: connect: Connection refused (error 111)
I20260928 04:58:34.964634 23164 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attempt: 1
I20260928 04:58:34.965183 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.965023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.0.1:33286)
0928 04:58:34.965090 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:34.965142 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 04:58:34.965687 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.965577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.0.1:33292)
0928 04:58:34.965631 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:34.965662 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 04:58:34.965840 23164 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39179: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39179: connect: Connection refused (error 111)
I20260928 04:58:34.984547 23089 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attempt: 1
I20260928 04:58:34.985157 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.984996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.0.1:33300)
0928 04:58:34.985071 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:34.985122 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 04:58:34.985584 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:34.985488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.0.1:33316)
0928 04:58:34.985537 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:34.985565 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 04:58:34.985693 23089 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39179: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39179: connect: Connection refused (error 111)
W20260928 04:58:35.022269 23171 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 04:58:35.022567 23171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:58:35.022588 23171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:58:35.022611 23171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:58:35.023952 23171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:58:35.024009 23171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:58:35.024024 23171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:58:35.024037 23171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:58:35.024049 23171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:58:35.025684 23171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37995,127.0.98.61:44347,127.0.98.60:39179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:58:35.025910 23171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:58:35.026095 23171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:58:35.031466 23177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:35.031466 23179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:58:35.031503 23176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:58:35.031656 23171 server_base.cc:1034] running on GCE node
I20260928 04:58:35.031988 23171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:58:35.032330 23171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:58:35.033504 23171 hybrid_clock.cc:648] HybridClock initialized: now 1790571515033480 us; error 43 us; skew 500 ppm
I20260928 04:58:35.033998 23164 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attempt: 2
I20260928 04:58:35.034454 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:35.034315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.0.1:33322)
0928 04:58:35.034384 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:35.034418 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 04:58:35.034839 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:35.034739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.0.1:33338)
0928 04:58:35.034788 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:35.034812 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 04:58:35.034962 23164 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39179: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39179: connect: Connection refused (error 111)
Sep 28 04:58:35 dist-test-slave-m39l krb5kdc[23008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:58:35.036588 23171 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 04:58:35.037127 23171 webserver.cc:460] Webserver started at http://127.0.98.60:45419/ using document root <none> and password file <none>
I20260928 04:58:35.037369 23171 fs_manager.cc:359] Metadata directory not provided
I20260928 04:58:35.037441 23171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:58:35.037595 23171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:58:35.038838 23171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-2/data/instance:
uuid: "dd9901d1476c4286af83ebe56e3aca96"
format_stamp: "Formatted at 2026-09-28 04:58:35 on dist-test-slave-m39l"
server_key: "47aa8363c26fe21dc031a7dd9fc4605f"
server_key_iv: "74341584e526494b16c877777e0b5aa0"
server_key_version: "encryptionkey@0"
I20260928 04:58:35.039230 23171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-2/wal/instance:
uuid: "dd9901d1476c4286af83ebe56e3aca96"
format_stamp: "Formatted at 2026-09-28 04:58:35 on dist-test-slave-m39l"
server_key: "47aa8363c26fe21dc031a7dd9fc4605f"
server_key_iv: "74341584e526494b16c877777e0b5aa0"
server_key_version: "encryptionkey@0"
I20260928 04:58:35.040663 23171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:58:35.041514 23186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:58:35.041744 23171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:58:35.041857 23171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-2/wal
uuid: "dd9901d1476c4286af83ebe56e3aca96"
format_stamp: "Formatted at 2026-09-28 04:58:35 on dist-test-slave-m39l"
server_key: "47aa8363c26fe21dc031a7dd9fc4605f"
server_key_iv: "74341584e526494b16c877777e0b5aa0"
server_key_version: "encryptionkey@0"
I20260928 04:58:35.041939 23171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:58:35.053898 23171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:58:35.056536 23171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:58:35.056738 23171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:58:35.060850 23089 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attempt: 2
I20260928 04:58:35.061015 23171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39179
I20260928 04:58:35.061097 23238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39179 every 8 connection(s)
I20260928 04:58:35.061537 23171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-2/data/info.pb
I20260928 04:58:35.062316 23239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:58:35.064248 23239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:58:35.067116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwNdUqW/build/release/bin/kudu as pid 23171
I20260928 04:58:35.067338 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwNdUqW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790571208412331-392-0/minicluster-data/master-2/wal/instance
I20260928 04:58:35.067550 392 external_mini_cluster.cc:1428] Setting key 6d80a949e845c837ea1b8df7b5ee4a75
Sep 28 04:58:35 dist-test-slave-m39l krb5kdc[23008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 04:58:35 dist-test-slave-m39l krb5kdc[23008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:58:35.072456 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:35.061180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.0.1:33346)
0928 04:58:35.061256 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:35.061271 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:35.061348 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:35.063037 (+ 1689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:35.063039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:35.063255 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:35.063308 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:35.063310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.064546 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.064547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:35.065283 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:35.065287 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.065918 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.065919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:35.065972 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:35.066569 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:35.066575 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:35.068322 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:35.070981 (+ 2659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:35.070982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:35.070983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:35.071217 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:35.071504 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:35.071505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:35.071506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:35.071590 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:35.072057 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:35.072059 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:35.072115 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 04:58:35.072219 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 04:58:35.072335 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 04:58:35.072707 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:35.061445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33346 (local address 127.0.98.60:39179)
0928 04:58:35.062156 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.062164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.062190 (+ 26us) server_negotiation.cc:407] Connection header received
0928 04:58:35.062827 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.062835 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.062888 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.063015 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.063456 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.064428 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.065367 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.065836 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.065947 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:35.068431 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:35.068438 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:35.068439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:35.068452 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:35.070888 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.071310 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.071314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.071315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.071434 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.071664 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.071665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.071666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.071971 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:35.072078 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:35.072297 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 04:58:35.072442 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":612,"threads_started":1}
I20260928 04:58:35.074549 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:35.064984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44974 (local address 127.0.98.62:37995)
0928 04:58:35.065032 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.065034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.065132 (+ 98us) server_negotiation.cc:407] Connection header received
0928 04:58:35.065202 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.065203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.065210 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.065257 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.065736 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.066406 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.067824 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.068121 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.068195 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:35.071144 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:35.071146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:35.071146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:35.071153 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:35.072984 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.073499 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.073500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.073501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.073566 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.073790 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.073791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.073791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.073991 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:35.074067 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:35.074316 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 04:58:35.074411 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 04:58:35 dist-test-slave-m39l krb5kdc[23008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:58:35.074922 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:35.064872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37995 (local address 127.0.0.1:44974)
0928 04:58:35.065041 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:35.065083 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:35.065148 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:35.065275 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:35.065277 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:35.065590 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:35.065655 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:35.065657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.066567 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.066568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:35.067672 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:35.067676 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.068295 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.068296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:35.068407 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:35.069072 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:35.069077 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:35.071049 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:35.073083 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:35.073088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:35.073090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:35.073372 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:35.073631 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:35.073632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:35.073633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:35.073707 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:35.074100 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:35.074107 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:35.074161 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 04:58:35.074659 (+ 498us) client_negotiation.cc:240] Negotiation successful
0928 04:58:35.074810 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
I20260928 04:58:35.075341 23089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad: Bootstrap starting.
I20260928 04:58:35.075481 23239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:58:35.076154 23089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:35.076502 23089 log.cc:824] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:35.077865 23089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad: No bootstrap required, opened a new log
I20260928 04:58:35.079952 23089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abace7644b640cd8501184b3fc02fad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } peers { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } } peers { permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } }
I20260928 04:58:35.080096 23089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abace7644b640cd8501184b3fc02fad, State: Initialized, Role: FOLLOWER
I20260928 04:58:35.080193 23089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abace7644b640cd8501184b3fc02fad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } peers { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } } peers { permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } }
I20260928 04:58:35.080677 23089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:35.080683 23246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abace7644b640cd8501184b3fc02fad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } peers { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } } peers { permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } } }
I20260928 04:58:35.080816 23246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 04:58:35 dist-test-slave-m39l krb5kdc[23008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260928 04:58:35.084067 23257 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:58:35.084133 23257 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:58:35.084901 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:35.068033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.0.98.60:39179)
0928 04:58:35.068192 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.068194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.068228 (+ 34us) server_negotiation.cc:407] Connection header received
0928 04:58:35.068315 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.068316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.068329 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.068401 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.069348 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.070071 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.071340 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.071674 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.071748 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:35.075311 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:35.075313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:35.075313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:35.075320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:35.083374 (+ 8054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.083837 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.083839 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.083839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.083916 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.084233 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.084233 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.084234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.084371 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:35.084581 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:35.084619 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 04:58:35.084752 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260928 04:58:35.085831 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:35.075769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44347 (local address 127.0.0.1:40890)
0928 04:58:35.076104 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:35.076116 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:35.076162 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:35.077033 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:35.077034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:35.077187 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:35.077262 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:35.077265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.078249 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.078250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:35.078974 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:35.078977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.079513 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.079514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:35.079587 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:35.080076 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:35.080082 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:35.081606 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:35.084485 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:35.084486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:35.084486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:35.084774 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:35.085133 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:35.085134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:35.085135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:35.085211 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:35.085591 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:35.085592 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:35.085631 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 04:58:35.085687 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 04:58:35.085769 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317}
I20260928 04:58:35.085938 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:35.076844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40890 (local address 127.0.98.61:44347)
0928 04:58:35.076916 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.076919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.076937 (+ 18us) server_negotiation.cc:407] Connection header received
0928 04:58:35.076961 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.076961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.076970 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.077023 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.077360 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.078136 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.079074 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.079420 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.079508 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:35.081698 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:35.081700 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:35.081700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:35.081709 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:35.084379 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.084853 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.084854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.084855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.084930 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.085310 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.085312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.085312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.085512 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:35.085600 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:35.085731 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 04:58:35.085840 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 04:58:35.086333 23239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:58:35.084030 23021 master_runner.cc:186] Error getting master registration for 127.0.98.61:44347: OK, instance_id { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" instance_seqno: 1790571514885900 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 04:58:35.087414 23021 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 04:58:35 dist-test-slave-m39l krb5kdc[23008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 04:58:35 dist-test-slave-m39l krb5kdc[23008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:58:35.094971 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:35.086705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.0.1:33378)
0928 04:58:35.086789 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:35.086800 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:35.086851 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:35.086974 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:35.086975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:35.087115 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:35.087146 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:35.087147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.087984 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.087986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:35.088858 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:35.088862 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.089344 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.089346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:35.089408 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:35.089950 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:35.089956 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:35.091878 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:35.093754 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:35.093756 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:35.093756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:35.093989 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:35.094190 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:35.094191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:35.094191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:35.094238 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:35.094592 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:35.094593 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:35.094648 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 04:58:35.094722 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 04:58:35.094816 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 04:58:35.095263 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:35.086739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33378 (local address 127.0.98.60:39179)
0928 04:58:35.086775 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.086777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.086834 (+ 57us) server_negotiation.cc:407] Connection header received
0928 04:58:35.086906 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.086907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.086917 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.086972 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.087216 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.087865 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.088953 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.089255 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.089322 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:35.092001 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:35.092003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:35.092003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:35.092011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:35.093648 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.094076 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.094077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.094078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.094128 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.094299 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.094299 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.094300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.094526 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:35.094600 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:35.095111 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 04:58:35.095193 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 04:58:35.096359 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:35.087783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.0.98.62:37995)
0928 04:58:35.087827 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.087829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.087838 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:58:35.087903 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.087904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.087913 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.087984 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.088491 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.089006 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.089898 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.090119 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.090213 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:35.092764 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:35.092766 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:35.092766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:35.092775 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:35.094256 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.094727 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.094728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.094728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.094792 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.095007 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.095008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.095008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.095158 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:35.095331 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:35.095897 (+ 566us) server_negotiation.cc:299] Negotiation successful
0928 04:58:35.096259 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:58:35.096830 23239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96: Bootstrap starting.
I20260928 04:58:35.097651 23239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:35.097995 23239 log.cc:824] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:35.098855 23239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96: No bootstrap required, opened a new log
I20260928 04:58:35.101020 23239 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abace7644b640cd8501184b3fc02fad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } peers { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } } peers { permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } }
I20260928 04:58:35.101207 23239 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd9901d1476c4286af83ebe56e3aca96, State: Initialized, Role: FOLLOWER
I20260928 04:58:35.101308 23239 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abace7644b640cd8501184b3fc02fad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } peers { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } } peers { permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } }
I20260928 04:58:35.101696 23239 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:35.101670 23260 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abace7644b640cd8501184b3fc02fad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } peers { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } } peers { permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } } }
I20260928 04:58:35.101787 23260 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 04:58:35 dist-test-slave-m39l krb5kdc[23008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260928 04:58:35.104287 23271 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:58:35.104456 23271 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:58:35.104969 23171 master_runner.cc:186] Error getting master registration for 127.0.98.61:44347: OK, instance_id { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" instance_seqno: 1790571514885900 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 04:58:35.107532 23171 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 04:58:35.108444 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:35.097035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40892 (local address 127.0.98.61:44347)
0928 04:58:35.097148 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.097150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.097157 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:58:35.097181 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.097182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.097192 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.097245 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.098002 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.098831 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.099809 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.100131 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.100209 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:35.103263 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:35.103265 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:35.103265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:35.103275 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:35.106658 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.107140 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.107142 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.107142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.107206 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.107401 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.107402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.107402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.107546 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:35.107725 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:35.108241 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 04:58:35.108328 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 04:58:35.109915 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:58:35.110900 23063 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9abace7644b640cd8501184b3fc02fad"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44976
I20260928 04:58:35.111095 23063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 04:58:35.111162 23063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:35.112300 23063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abace7644b640cd8501184b3fc02fad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } peers { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } } peers { permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } }
I20260928 04:58:35.112605 23063 leader_election.cc:288] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [CANDIDATE]: Term 1 election: Requested vote from peers c02510c1bb304159b77156b7ac46eec1 (127.0.98.61:44347), dd9901d1476c4286af83ebe56e3aca96 (127.0.98.60:39179)
I20260928 04:58:35.113008 23213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9abace7644b640cd8501184b3fc02fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dd9901d1476c4286af83ebe56e3aca96"
I20260928 04:58:35.113199 23213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:35.114153 23213 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9abace7644b640cd8501184b3fc02fad in term 1.
I20260928 04:58:35.114455 23038 leader_election.cc:302] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9abace7644b640cd8501184b3fc02fad, dd9901d1476c4286af83ebe56e3aca96; no voters:
I20260928 04:58:35.114662 23246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:58:35.114892 23246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [term 1 LEADER]: Becoming Leader. State: Replica: 9abace7644b640cd8501184b3fc02fad, State: Running, Role: LEADER
I20260928 04:58:35.115011 23246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abace7644b640cd8501184b3fc02fad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } peers { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } } peers { permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } }
I20260928 04:58:35.115504 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9abace7644b640cd8501184b3fc02fad. Latest consensus state: current_term: 1 leader_uuid: "9abace7644b640cd8501184b3fc02fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abace7644b640cd8501184b3fc02fad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } peers { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } } peers { permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } } }
I20260928 04:58:35.115724 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:35.120504 23279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:58:35.120786 23279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:58:35.126104 23164 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attempt: 3
I20260928 04:58:35.129192 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:35.119252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.0.98.60:39179)
0928 04:58:35.119313 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.119316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.119744 (+ 428us) server_negotiation.cc:407] Connection header received
0928 04:58:35.119811 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.119813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.119825 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.119897 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.120724 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.121455 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.122584 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.122918 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.122971 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:35.125338 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:35.125340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:35.125340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:35.125349 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:35.127152 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.127760 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.127761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.127761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.127842 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.128177 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.128178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.128178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.128315 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:35.128391 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:35.128974 (+ 583us) server_negotiation.cc:299] Negotiation successful
0928 04:58:35.129078 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:58:35.131300 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:35.118942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.0.98.61:44347)
0928 04:58:35.119009 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.119011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.119605 (+ 594us) server_negotiation.cc:407] Connection header received
0928 04:58:35.119678 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.119680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.119693 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.119775 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.121537 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.122336 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.124332 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.124712 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.124760 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:35.127154 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:35.127155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:35.127156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:35.127164 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:35.129118 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.129803 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.129804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.129805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.129876 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.130111 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.130112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.130112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.130253 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:35.130757 (+ 504us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:35.131058 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 04:58:35.131158 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 04:58:35 dist-test-slave-m39l krb5kdc[23008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:58:35.135793 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:35.126425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.0.1:33398)
0928 04:58:35.126562 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 04:58:35.126575 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:58:35.126626 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:58:35.126939 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:58:35.126939 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:58:35.127136 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:58:35.127188 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:35.127190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.128323 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.128325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:35.129442 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:58:35.129445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.130084 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.130085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:58:35.130152 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:35.130704 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:58:35.130709 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:58:35.132374 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:58:35.134465 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:35.134466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:35.134467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:35.134762 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:35.135011 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:58:35.135011 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:58:35.135012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:58:35.135091 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:58:35.135438 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:58:35.135439 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:58:35.135492 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 04:58:35.135554 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 04:58:35.135671 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260928 04:58:35.136368 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:35.126718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.0.98.60:39179)
0928 04:58:35.126767 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.126769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.126777 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:58:35.126804 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.126805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.126818 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.126875 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.127281 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.128206 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.129546 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.129957 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.130024 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:35.132490 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:35.132492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:35.132492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:35.132500 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:35.134339 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.134854 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.134856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.134856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.134919 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.135172 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.135173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.135173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.135335 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:35.135408 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:35.136156 (+ 748us) server_negotiation.cc:299] Negotiation successful
0928 04:58:35.136252 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 04:58:35.138159 23164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1: Bootstrap starting.
I20260928 04:58:35.138581 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:35.119513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44990 (local address 127.0.98.62:37995)
0928 04:58:35.119558 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.119560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.119896 (+ 336us) server_negotiation.cc:407] Connection header received
0928 04:58:35.119997 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.119999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.120011 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.120043 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.130585 (+ 10542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.131314 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.132327 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:35.132677 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:35.132757 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:35.134816 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:58:35.134817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:58:35.134818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:58:35.134826 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:58:35.136751 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.137351 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.137352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.137352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.137426 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:58:35.137657 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:58:35.137657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:58:35.137658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:58:35.137807 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:58:35.137899 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:35.138342 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 04:58:35.138423 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:58:35.139043 23164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1: Neither blocks nor log segments found. Creating new log.
I20260928 04:58:35.139427 23164 log.cc:824] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1: Log is configured to *not* fsync() on all Append() calls
I20260928 04:58:35.139665 23213 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96 [term 1 FOLLOWER]: Refusing update from remote peer 9abace7644b640cd8501184b3fc02fad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:58:35.140023 23246 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:58:35.141047 23164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1: No bootstrap required, opened a new log
I20260928 04:58:35.142280 23260 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9abace7644b640cd8501184b3fc02fad. Latest consensus state: current_term: 1 leader_uuid: "9abace7644b640cd8501184b3fc02fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abace7644b640cd8501184b3fc02fad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } peers { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } } peers { permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } } }
I20260928 04:58:35.142380 23260 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:58:35.143957 23164 raft_consensus.cc:348] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abace7644b640cd8501184b3fc02fad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } peers { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } } peers { permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } }
I20260928 04:58:35.144111 23164 raft_consensus.cc:729] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c02510c1bb304159b77156b7ac46eec1, State: Initialized, Role: FOLLOWER
I20260928 04:58:35.144202 23164 consensus_queue.cc:260] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abace7644b640cd8501184b3fc02fad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } peers { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } } peers { permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } }
I20260928 04:58:35.144605 23164 sys_catalog.cc:564] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:58:35.144706 23137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:58:35.145720 23137 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1 [term 1 FOLLOWER]: Refusing update from remote peer 9abace7644b640cd8501184b3fc02fad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:58:35.145932 23138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9abace7644b640cd8501184b3fc02fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c02510c1bb304159b77156b7ac46eec1"
I20260928 04:58:35.146219 23277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:58:35.147105 23138 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9abace7644b640cd8501184b3fc02fad in term 1.
W20260928 04:58:35.149286 23301 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:58:35.149422 23301 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:58:35.149855 23298 sys_catalog.cc:455] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9abace7644b640cd8501184b3fc02fad. Latest consensus state: current_term: 1 leader_uuid: "9abace7644b640cd8501184b3fc02fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abace7644b640cd8501184b3fc02fad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } peers { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } } peers { permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } } }
I20260928 04:58:35.149974 23298 sys_catalog.cc:458] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:58:35.150576 23283 mvcc.cc:205] Tried to move back new op lower bound from 7334180926008909824 to 7334180925911986176. Current Snapshot: MvccSnapshot[applied={T|T < 7334180926008909824}]
I20260928 04:58:35.150875 23246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9abace7644b640cd8501184b3fc02fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abace7644b640cd8501184b3fc02fad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } peers { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } } peers { permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } } }
I20260928 04:58:35.151001 23246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:35.150936 23279 catalog_manager.cc:1347] Generated new cluster ID: 64498b1aa3734cfdb208251dffe0f4d2
I20260928 04:58:35.151156 23279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:58:35.152153 23260 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9abace7644b640cd8501184b3fc02fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abace7644b640cd8501184b3fc02fad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } peers { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } } peers { permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } } }
I20260928 04:58:35.152251 23260 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:58:35.150429 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9abace7644b640cd8501184b3fc02fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abace7644b640cd8501184b3fc02fad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } peers { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } } peers { permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } } }
I20260928 04:58:35.153023 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad [sys.catalog]: This master's current role is: LEADER
I20260928 04:58:35.144589 23287 sys_catalog.cc:455] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abace7644b640cd8501184b3fc02fad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } peers { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } } peers { permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } } }
I20260928 04:58:35.154135 23287 sys_catalog.cc:458] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:58:35.154196 23287 sys_catalog.cc:455] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9abace7644b640cd8501184b3fc02fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abace7644b640cd8501184b3fc02fad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } peers { permanent_uuid: "c02510c1bb304159b77156b7ac46eec1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44347 } } peers { permanent_uuid: "dd9901d1476c4286af83ebe56e3aca96" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } } }
I20260928 04:58:35.154273 23287 sys_catalog.cc:458] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:58:35.194813 23279 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:58:35.195662 23279 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:58:35.205170 23279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9abace7644b640cd8501184b3fc02fad: Generated new TSK 0
I20260928 04:58:35.205476 23279 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 04:58:35.236270 23092 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:35.234665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44996 (local address 127.0.98.62:37995)
0928 04:58:35.234719 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.234721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.235479 (+ 758us) server_negotiation.cc:407] Connection header received
0928 04:58:35.235598 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.235599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.235613 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.235645 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.236193 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44996: BlockingRecv error: recv got EOF from 127.0.0.1:44996 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 04:58:35.237140 23241 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:35.235171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.0.98.60:39179)
0928 04:58:35.235217 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.235219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.236554 (+ 1335us) server_negotiation.cc:407] Connection header received
0928 04:58:35.236660 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.236661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.236674 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.236705 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.237071 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33404: BlockingRecv error: recv got EOF from 127.0.0.1:33404 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 04:58:35.238302 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:35.237557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40920 (local address 127.0.98.61:44347)
0928 04:58:35.237601 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.237602 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.237729 (+ 127us) server_negotiation.cc:407] Connection header received
0928 04:58:35.237825 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.237826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.237846 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.237886 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.238233 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40920: BlockingRecv error: recv got EOF from 127.0.0.1:40920 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 04:58:35.241052 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:35.240094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.0.98.61:44347)
0928 04:58:35.240137 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.240138 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.240338 (+ 200us) server_negotiation.cc:407] Connection header received
0928 04:58:35.240461 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.240463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.240484 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.240516 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.240999 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40922: BlockingRecv error: recv got EOF from 127.0.0.1:40922 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 04:58:35.244549 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:35.243518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40932 (local address 127.0.98.61:44347)
0928 04:58:35.243560 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.243562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.243813 (+ 251us) server_negotiation.cc:407] Connection header received
0928 04:58:35.243938 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.243940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.243954 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.243987 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.244489 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40932: BlockingRecv error: recv got EOF from 127.0.0.1:40932 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 04:58:35.248700 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:35.247825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40942 (local address 127.0.98.61:44347)
0928 04:58:35.247883 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.247885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.248127 (+ 242us) server_negotiation.cc:407] Connection header received
0928 04:58:35.248232 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.248233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.248245 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.248276 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.248641 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40942: BlockingRecv error: recv got EOF from 127.0.0.1:40942 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 04:58:35.255920 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:35.254449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40958 (local address 127.0.98.61:44347)
0928 04:58:35.254695 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.254698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.255020 (+ 322us) server_negotiation.cc:407] Connection header received
0928 04:58:35.255137 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.255139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.255152 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.255184 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.255855 (+ 671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40958: BlockingRecv error: recv got EOF from 127.0.0.1:40958 (error 108)
Metrics: {"server-negotiator.queue_time_us":217}
W20260928 04:58:35.266887 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:35.265752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40966 (local address 127.0.98.61:44347)
0928 04:58:35.265802 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.265805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.266121 (+ 316us) server_negotiation.cc:407] Connection header received
0928 04:58:35.266230 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.266232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.266245 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.266274 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.266810 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40966: BlockingRecv error: recv got EOF from 127.0.0.1:40966 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 04:58:35.286023 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:35.284863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40980 (local address 127.0.98.61:44347)
0928 04:58:35.284912 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.284915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.285218 (+ 303us) server_negotiation.cc:407] Connection header received
0928 04:58:35.285358 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.285359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.285373 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.285408 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.285953 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40980: BlockingRecv error: recv got EOF from 127.0.0.1:40980 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 04:58:35.320683 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:35.319775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.0.98.61:44347)
0928 04:58:35.319832 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.319834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.320021 (+ 187us) server_negotiation.cc:407] Connection header received
0928 04:58:35.320084 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.320085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.320100 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.320188 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.320601 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40988: BlockingRecv error: recv got EOF from 127.0.0.1:40988 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 04:58:35.387167 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:35.386175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40998 (local address 127.0.98.61:44347)
0928 04:58:35.386274 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.386276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.386432 (+ 156us) server_negotiation.cc:407] Connection header received
0928 04:58:35.386570 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.386571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.386584 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.386637 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.387094 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40998: BlockingRecv error: recv got EOF from 127.0.0.1:40998 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 04:58:35.517526 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:35.516563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41002 (local address 127.0.98.61:44347)
0928 04:58:35.516663 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.516666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.516764 (+ 98us) server_negotiation.cc:407] Connection header received
0928 04:58:35.516935 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.516936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.516948 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.517030 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.517418 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41002: BlockingRecv error: recv got EOF from 127.0.0.1:41002 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 04:58:35.776031 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:58:35.774918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41018 (local address 127.0.98.61:44347)
0928 04:58:35.775036 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:35.775039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:35.775324 (+ 285us) server_negotiation.cc:407] Connection header received
0928 04:58:35.775444 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:35.775446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:35.775460 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:35.775555 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:58:35.775935 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41018: BlockingRecv error: recv got EOF from 127.0.0.1:41018 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 04:58:36.105012 23271 catalog_manager.cc:1259] Loaded cluster ID: 64498b1aa3734cfdb208251dffe0f4d2
I20260928 04:58:36.105130 23271 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96: loading cluster ID for follower catalog manager: success
I20260928 04:58:36.106001 23271 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96: acquiring CA information for follower catalog manager: success
I20260928 04:58:36.106348 23271 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dd9901d1476c4286af83ebe56e3aca96: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:58:36.149958 23301 catalog_manager.cc:1259] Loaded cluster ID: 64498b1aa3734cfdb208251dffe0f4d2
I20260928 04:58:36.150074 23301 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1: loading cluster ID for follower catalog manager: success
I20260928 04:58:36.150967 23301 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1: acquiring CA information for follower catalog manager: success
I20260928 04:58:36.151304 23301 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c02510c1bb304159b77156b7ac46eec1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:58:36.292428 23371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:36.289383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41022 (local address 127.0.98.61:44347)
0928 04:58:36.289541 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:36.289543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:36.289639 (+ 96us) server_negotiation.cc:407] Connection header received
0928 04:58:36.289742 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:36.289743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:36.289816 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:36.289916 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:58:36.290095 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:36.290705 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:36.291517 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:36.291735 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:36.291817 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:36.292015 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:58:36.292139 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:58:36.292228 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:36.292283 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 04:58:36.292317 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260928 04:58:36.297147 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:36.294543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33406 (local address 127.0.98.60:39179)
0928 04:58:36.294664 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:36.294666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:36.294676 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:58:36.294765 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:36.294766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:36.294803 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:36.294891 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:58:36.295040 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:36.295531 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:36.296373 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:36.296567 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:36.296617 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:36.296795 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:58:36.296909 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:58:36.296966 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:36.297004 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 04:58:36.297024 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260928 04:58:36.302376 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:36.299128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45000 (local address 127.0.98.62:37995)
0928 04:58:36.299288 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:36.299290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:36.299332 (+ 42us) server_negotiation.cc:407] Connection header received
0928 04:58:36.299431 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:36.299432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:36.299492 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:36.299581 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:58:36.299750 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:36.300530 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:36.301363 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:36.301646 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:36.301713 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:36.301911 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:58:36.302053 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:58:36.302134 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:36.302222 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 04:58:36.302257 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260928 04:58:36.302471 23371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:36.299075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41034 (local address 127.0.98.61:44347)
0928 04:58:36.299261 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:36.299264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:36.299536 (+ 272us) server_negotiation.cc:407] Connection header received
0928 04:58:36.299600 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:36.299601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:36.299614 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:36.299665 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:58:36.299840 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:36.300331 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:36.301568 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:36.301898 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:36.301976 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:36.302170 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:58:36.302208 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:58:36.302258 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:36.302349 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 04:58:36.302374 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260928 04:58:36.302928 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:58:36.299271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.0.98.60:39179)
0928 04:58:36.299314 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:58:36.299315 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:58:36.299363 (+ 48us) server_negotiation.cc:407] Connection header received
0928 04:58:36.299436 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:58:36.299437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:58:36.299447 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:58:36.299504 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:58:36.299692 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:36.300481 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:36.302023 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:58:36.302353 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:58:36.302413 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:58:36.302644 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:58:36.302680 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:58:36.302735 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:58:36.302813 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 04:58:36.302837 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:58:36.305047 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 23021
I20260928 04:58:36.310731 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 23096
I20260928 04:58:36.315548 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwNdUqW/build/release/bin/kudu with pid 23171
2026-09-28T04:58:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2962 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6094 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (307909 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS